{"id":"3fdb2437-61b8-4c36-bb9f-166c53c26c54","arxiv_id":"2501.05874","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"VideoRAG retrieves query-relevant videos from a corpus and uses large video-language models to generate answers grounded in the videos' visual frames and transcripts.","lead":"VideoRAG is a system that answers questions by first searching a large video library for relevant clips, then feeding the best clip's frames and transcript into a video-language model to generate the answer. It is an early RAG pipeline that retrieves videos dynamically and uses both what is seen and what is said in the video.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains over TextVideoRAG conflate visual grounding with a different, possibly stronger retriever; no condition holds retrieval fixed while varying generation modality, so the central claim about visual video information is not isolated.","rationale":"The reader's conditional verdict is appropriate. The paper's strongest quantitative evidence is Table 1, where VideoRAG beats TextVideoRAG by roughly 2 ROUGE-L points with LLaVA-Video and larger margins with InternVL2.5. But TextVideoRAG and VideoRAG differ in both the retriever and the generation input. Table 2 demonstrates retrieval quality depends on modality (text R@1 0.088 > visual 0.054), so the two systems are not answering from the same evidence. Table 4's ablation also changes two variables at once. A matched-retrieval experiment would settle whether the advantage comes from the visual modality or simply from a better retriever. I do not see a more fundamental flaw: the framework is coherent, code is released, results are consistent across three LVLMs and two datasets, and the human evaluation (though small) aligns with automatic metrics. The absent retrieval control is the single load-bearing gap and explains the CONDITIONAL verdict; no verdict change is needed.","tokens_in":24303,"tokens_out":5766,"duration_ms":58899,"concrete_test":"On the WikiHowQA+HowTo100M benchmark, fix a single retriever (e.g., InternVideo2 ensemble, top-1) and a single LVLM, and generate from (a) transcript-only, (b) frames-only, (c) frames+transcript; also run TextVideoRAG under the same fixed retriever. Compare ROUGE-L, BLEU-4, BERTScore, and G-Eval over at least 3 seeds, reporting mean±std and retrieval R@1 per condition. If transcript-only matches or beats frames+transcript with retrieval held fixed, the improvement is not from visual grounding; if frames-only beats transcript-only on identical retrieved videos, the paper's video-visual claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—VideoRAG significantly surpasses text-video RAG baselines—rests on Table 1, but the comparison does not isolate the mechanism. TextVideoRAG differs from VideoRAG in two coupled ways: (i) retrieval uses a different pipeline (text-only indexing/encoding rather than VideoRAG's InternVideo2 ensemble with adaptive frame selection) and (ii) generation consumes only transcripts instead of frames or frames+transcripts. Table 2 shows retrieval quality is modality-dependent (textual R@1 0.088 vs. visual 0.054, ensemble 0.103), so the two conditions start from different retrieved videos. Table 4's ablation changes both the input modality and the retriever, so it cannot separate 'better retrieval' from 'better grounding'. Thus the observed gains could be due to retrieving more useful content rather than to the visual/audio-visual grounding that the paper claims as its contribution. This is not a conceptual flaw, but it is load-bearing because the paper's novelty claim is specifically about using visual video information; if the gain vanishes once retrieval is matched, the main conclusion is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VideoRAG, a retrieval-augmented generation framework that operates over a video corpus. For a given query, it retrieves videos using LVLM-based embeddings that combine visual frames (selected by a trained adaptive frame selector) and textual transcripts (original subtitles or ASR-generated auxiliary text), then feeds the retrieved videos jointly with the query into an LVLM to generate answers. The authors construct evaluation sets from WikiHowQA with HowTo100M and a synthetic QA dataset, and compare against textual, text-image, and text-video RAG baselines across three LVLMs (LLaVA-Video, InternVL2.5, Qwen2.5-VL). Results in Table 1 show VideoRAG outperforming all baselines on ROUGE-L, BLEU-4, BERTScore, and G-Eval, with a supporting human evaluation. The paper also reports retrieval ablations, frame-selection comparisons, and category-wise analyses.","tokens_in":24494,"tokens_out":4145,"duration_ms":42004,"significance":"If the reported gains are attributable to the joint use of visual and textual video content, this is a valuable step toward extending RAG to video corpora, which are rich but underexplored as knowledge sources. The paper ships code, evaluates across multiple LVLMs and metrics, includes a human study corroborating the automatic results, and provides detailed ablations of retrieval modalities and frame-selection strategies. These are concrete strengths. However, the central claim that visual grounding drives the improvement is not yet isolated from retrieval-quality differences, so the significance of the specific mechanism remains conditional on additional controlled experiments.","major_comments":[{"comment":"The main comparison between VideoRAG and TEXTVIDEORAG changes two factors simultaneously: the retriever (InternVideo2 with ensemble features and adaptive frame selection versus text-only indexing/encoding) and the generation input modality (frames versus text-only). Because Table 2 shows that the ensemble retriever has higher recall (R@1 0.103) than textual features alone (R@1 0.088), the improved generation results may be driven by retrieving more relevant videos rather than by the visual grounding that the paper claims as its contribution. Without a condition that holds retrieval fixed while varying the generation modality (e.g., generating from the same top-1 video with text-only versus frames-only versus frames+text), the central claim about visual video information is not isolated.","section":"Section 3.2, Table 1"},{"comment":"The paper repeatedly uses the word 'significantly' to describe the improvements of VideoRAG over baselines, but no error bars, confidence intervals, or significance tests are reported. Several gaps are small (e.g., for LLaVA-Video, VideoRAG-V and VideoRAG-VT differ by 0.02 ROUGE-L on WikiHowQA), and the magnitudes vary across LVLMs and datasets. The authors should report variances (e.g., bootstrap confidence intervals or multiple random seeds) and perform paired significance tests for the key pairwise comparisons to support the claimed significance.","section":"Section 3.1 / Table 1"},{"comment":"The adaptive frame selectors are trained with labels derived from the same metrics used in evaluation: the retrieval selector uses retrieval-similarity-based labels and the generation selector uses ROUGE-L-based labels, while the reported improvements (Table 3) are measured on retrieval recall and ROUGE-L. This creates a risk that the gains over uniform sampling reflect direct optimization of the evaluation objectives rather than generally informative frame selection. The authors should either evaluate the selected frames on a metric not used for training (e.g., human ratings or G-Eval conditioned on frame choice) or discuss this potential inflation and provide evidence that the selection transfers to held-out metrics.","section":"Section 2.3 / Appendix A.3, Table 3"}],"minor_comments":[{"comment":"It is unclear how the WikiHowQA queries are paired with HowTo100M videos for the Oracle condition and how the retrieval corpus is constructed; please clarify the association procedure and whether the corpus contains all videos or a filtered subset.","section":"Section 3.1, Datasets"},{"comment":"The notation Comb(·) is described as randomly sampling a subset of m frames from n, but the method first applies k-means++ clustering to reduce the frame space; please clarify how Comb operates on the reduced candidate set and whether the clustering step is included in the theoretical formulation.","section":"Section 2.3, Eq. (1)-(2)"},{"comment":"The paper states that 40 random subsets are sampled from the 64C32 combinations, but no rationale is given for this number; a short justification or sensitivity analysis would be helpful.","section":"Appendix A.3, Generation frame selection"},{"comment":"The human evaluation description does not specify how many evaluators assessed each query, how individual Likert scores were aggregated into the reported mean, or how the inter-annotator correlation was computed; please provide this detail.","section":"Table 6"},{"comment":"The x-axis label appears truncated as 'Combination Ratio ()'; the symbol should be written as α and defined in the caption.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The core idea is solid and the paper is well positioned for the multimodal RAG community, but the experimental design currently conflates retrieval quality with generation modality. The authors should be asked to add a controlled condition that fixes the retrieved videos across the text-only and video-input generation settings, and to provide statistical support for the 'significant' claims. With those additions, the paper could become a strong contribution; without them, the main contribution is not convincingly established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"VideoRAG is a solid systems paper and worth a real referee. It does something new: it puts together corpus-level video retrieval, LVLM-based generation over frames and transcripts, and a learned frame selector, and it ships code. The gains over text-only and text-image RAG are consistent across three LVLMs and several metrics, and the small human evaluation agrees. That is real evidence.\n\nThe paper also does a few things right. It is clear about the components, uses off-the-shelf encoders and LVLMs, and honestly reports retrieval quality by modality (Table 2). The ASR fallback for missing subtitles is a practical, sensible addition. The frame-selection ablation shows adaptive selection beats uniform sampling.\n\nThe soft spot is the one the stress-test flags, and I think it is legitimate. The main comparison changes two things at once: retrieval pipeline and generation input. TextVideoRAG retrieves with text-only indexing and generates from transcripts; VideoRAG retrieves with InternVideo2 plus adaptive frame selection and generates from frames. Table 2 shows retrieval quality varies a lot by feature type (textual R@1 0.088, visual 0.054, ensemble 0.103), so the two conditions are starting from different videos. There is no condition that holds retrieval fixed and varies only whether generation consumes frames or transcripts. That means the reported gains could be driven by better retrieval rather than by visual grounding, which is the paper's specific novelty claim. This does not kill the paper—the system still appears to work better than baselines—but it does mean the headline interpretation is not fully supported.\n\nA related smaller issue: the frame selector for generation is trained to optimize ROUGE-L, and the retrieval selector is trained on retrieval similarity, both of which are used as evaluation metrics. That is a mild circularity for the frame-selection component, not for the central claim, and it is worth noting but not fatal.\n\nThe lack of error bars or significance tests is also a weakness, but the consistency across three LVLMs mitigates it somewhat.\n\nBottom line: this is a useful contribution to multimodal RAG, the code is out, and the questions I raised are answerable with one controlled experiment plus standard errors. I would send it to review and ask for that experiment. A reader who works on RAG or video understanding will get value from it.","headline":"VideoRAG is a solid integration of video retrieval, LVLM generation, and frame selection, but the experiments don't isolate visual grounding from retrieval quality, so the headline claim is suggestive, not proven.","tokens_in":25071,"tokens_out":2182,"would_cite":true,"duration_ms":21778,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Retrieving videos as knowledge improves RAG answers over text-only and image-based retrieval.","keywords":["retrieval-augmented generation","video retrieval","large video language models","multimodal knowledge","frame selection","question answering","HowTo100M","automatic speech recognition"],"falsifier":"An experiment that retrieves with a single fixed retriever and then generates from transcript only, frames only, and frames plus transcript for the same retrieved videos would isolate the visual contribution; if frames plus transcript does not beat transcript only, the central claim fails. A second check would swap VideoRAG's video-specific retrieval encoder for a text retriever while keeping the generation input identical, to show how much of the gap comes from retrieval rather than modality fusion.","tokens_in":24063,"feed_emoji":"📹","tokens_out":7610,"duration_ms":68145,"temperature":0.7,"pith_summary":"VideoRAG asks whether a video corpus can serve as external knowledge for retrieval-augmented generation, and answers yes. Instead of flattening videos into text or assuming the relevant video is already known, the method retrieves videos by embedding queries and video frames plus transcripts through a large video-language model, then feeds the retrieved video's most informative frames and text into the same model to generate the answer. The paper reports that this joint visual-textual video pipeline outperforms text-only, text-image, and text-video RAG baselines on instructional question answering built from WikiHowQA and HowTo100M. It also introduces a trained frame-selection step that beats uniform sampling, and uses speech-to-text to supply transcripts when subtitles are missing. If correct, the work extends the RAG paradigm from text and images to open-domain video knowledge.","feed_headline":"Video retrieval beats text-only RAG on how-to questions","feed_subtitle":"Answering with frames and transcripts from retrieved videos outdoes text, image, and text-video baselines.","key_machinery":"The load-bearing object is the large video-language model (LVLM), a model that encodes video frames and text into a shared token stream and can both embed a query-video pair for retrieval and generate an answer from the concatenation of retrieved frames, transcripts, and query. Around it sit two supporting mechanisms: a learned frame-scoring function that scores subsets of frames, with k-means++ first reducing the search space, and picks the most informative frames separately for retrieval and generation; and an automatic-speech-recognition fallback that produces auxiliary text for videos without subtitles. These allow the pipeline to treat the video itself as the knowledge unit rather than a pre-chosen video or a text summary.","core_discovery":"The paper is trying to establish that a video corpus can be a practical external knowledge source for retrieval-augmented generation when both retrieval and generation are handled by a large video-language model. Concretely, it claims that dynamically retrieving videos by embedding similarity and then generating from selected frames plus transcripts yields better answers on instructional question answering than retrieving text documents, text-image pairs, or text-only video descriptions. The experiments compare six baselines and report VideoRAG as the best on ROUGE-L, BLEU-4, BERTScore, and G-Eval across three LVLMs, with an oracle setting showing that better retrieval would help further. The paper also claims its adaptive frame selection improves both retrieval and generation relative to uniform sampling, and that automatic-speech-recognition-derived transcripts stand in for missing subtitles.","pith_inferences":["The paper's evaluation is instructional; the same pipeline may transfer to news, sports, or documentary video, where the visual signal carries information absent from transcripts, but that transfer is not tested here.","Because the baselines and VideoRAG differ in both retrieval encoder and generation input, part of the reported gap could come from retrieval quality rather than joint visual-textual generation; a retrieval-held-fixed ablation would settle this.","The frame-selection scorer is trained on downstream retrieval and generation success labels, so it could be retrained for other video tasks, such as video grounding or long-video QA, with the same label-generation recipe.","A direct prediction of the visual-value claim is that on queries whose answers live only in the image, such as how a dog's expression changes, the gap over transcript-only RAG should be larger than on text-redundant queries."],"forward_implications":["For how-to question answering, a system can answer without knowing in advance which video is relevant, because retrieval happens dynamically over the video corpus.","Answers grounded in video frames plus transcripts surpass answers grounded in transcripts alone, so video corpora should be treated as first-class RAG knowledge sources.","Selecting a small number of informative frames is enough to beat uniform sampling, which lowers the cost of processing long videos.","Videos without subtitles can still contribute textual signal by running automatic speech recognition, making the approach applicable to raw video collections.","The method scales with LVLM size and holds across query categories, so larger video-language models are a direct lever for further gains."],"supporting_citations":[{"why":"Supplies the HowTo100M video corpus of instructional YouTube videos that VideoRAG retrieves from.","marker":"Miech et al., 2019"},{"why":"Supplies the WikiHowQA query-answer pairs used to build the RAG evaluation set.","marker":"Bolotova-Baranova et al., 2023"},{"why":"Provides the InternVideo2 video and text encoders used for query-video retrieval.","marker":"Wang et al., 2024c"},{"why":"Provides the LLaVA-Video LVLM used as a generation backbone for video-augmented answering.","marker":"Zhang et al., 2024c"},{"why":"Provides CLIP features and the scoring backbone used in the frame-selection mechanism.","marker":"Radford et al., 2021"},{"why":"Provides Whisper, used to generate auxiliary text from audio when subtitles are unavailable.","marker":"Radford et al., 2023"},{"why":"Defines the BM25 text-retrieval baseline that VideoRAG is compared against.","marker":"Robertson et al., 1994"},{"why":"Defines the DPR dense text-retrieval baseline that VideoRAG is compared against.","marker":"Karpukhin et al., 2020"},{"why":"Defines the text-image multimodal RAG baseline that VideoRAG is compared against.","marker":"Chen et al., 2022"},{"why":"Represents the text-video RAG baseline that VideoRAG extends and surpasses.","marker":"Arefeen et al., 2024"}],"fun_headline_variants":["VideoRAG retrieves videos to answer how-to questions better","Video retrieval + LVLM beats text-only RAG on how-to tasks","Dynamic video retrieval improves RAG answers on instructional queries","VideoRAG: RAG with video retrieval and frame selection wins","Videos as knowledge: VideoRAG outperforms text RAG on how-to QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes the reported gains come from jointly using visual and textual video content, even though the video pipeline also uses a different retrieval encoder than the text baselines and the retrieval step is never held fixed while only the generation modality changes.","fun_headline_variants_meta":{"raw":{"variants":["VideoRAG retrieves videos to answer how-to questions better","Video retrieval + LVLM beats text-only RAG on how-to tasks","Dynamic video retrieval improves RAG answers on instructional queries","VideoRAG: RAG with video retrieval and frame selection wins","Videos as knowledge: VideoRAG outperforms text RAG on how-to QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000161,"raw_usage":{"total_tokens":1252,"prompt_tokens":977,"completion_tokens":275,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":183}},"tokens_in":593,"tokens_out":275,"duration_ms":3065,"temperature":1.0,"reasoning_tokens":183,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:05:55.928854+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An experiment that retrieves with a single fixed retriever and then generates from transcript only, frames only, and frames plus transcript for the same retrieved videos would isolate the visual contribution; if frames plus transcript does not beat transcript only, the central claim fails. A second check would swap VideoRAG's video-specific retrieval encoder for a text retriever while keeping the generation input identical, to show how much of the gap comes from retrieval rather than modality fusion.","supporting_citations":[],"review_version":1}