{"id":"d1d0f38e-6704-4cb1-a2a7-53fdc4262a95","arxiv_id":"2608.03979","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Video-DeepResearch trains video agents with a two-stage SFT plus GRPO pipeline to do visual web searches before text searches, and introduces a 200-question benchmark requiring both video grounding and open-web knowledge.","lead":"This paper introduces Video-DeepResearch, a system that teaches AI agents to search the web using what they see in videos, not just text. It also introduces a benchmark showing that current top models often skip looking at video evidence and instead answer from memory.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's 'provably requires tools' guarantee rests on a single-model four-rollout filter; the 5-point SOTA gap loses meaning if any test items overlap training data or are answerable from parametric memory.","rationale":"I read the paper as proposing a training pipeline and a benchmark, with the headline empirical claim being the SOTA accuracy. The load-bearing assumption, as the reader identified, is that the benchmark is leakage-free and that 'provably requires tools' is true. My assessment agrees: the filtering procedure is empirical, not a proof, and the absence of a reported contamination analysis leaves a real risk. The 100-vs-200 size inconsistency is a separate but related fragility: if the benchmark is 100 items, the 5-point gap is not statistically meaningful. I did not find an internal inconsistency beyond these issues; the pipeline is coherent and the ablations support the training contribution. Therefore the appropriate verdict remains CONDITIONAL, contingent on releasing data and performing the leakage/contamination checks. I see no basis for rejection, because the method and data engine are plausibly useful regardless of the exact SOTA claim.","tokens_in":19080,"tokens_out":6206,"duration_ms":63291,"concrete_test":"Release VIDEODR-BENCH and the training set. (1) Run exact and near-duplicate matching (video IDs, frames, and question text) between the benchmark and the 30K/7K training data; report overlap rate. (2) Run each benchmark question through the four tool-free rollouts with a diverse set of strong models (GPT-5, Claude-4.5-Sonnet, Gemini 2.5 Pro, Qwen3.5-397B) using the same judge; if any model answers >10% of questions with zero tool calls, the 'provably requires tools' claim is falsified. (3) If feasible, have human annotators independently label whether each question can be answered from the video alone or from general knowledge.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (64.0% vs 59.0% SOTA) and the benchmark's stated property that 'every question provably requires both visual search and external knowledge reasoning' depend on the leakage filter in Sec 3.1 Step 2 and Sec 3.4. That filter is not a proof: four tool-free rollouts of one model (Qwen3.5-397B-A17B) only show that that model did not answer correctly; it says nothing about GPT-5, Claude-4.5-Sonnet, or the trained model's parametric memory. Moreover, the benchmark is sampled from the same curated video pool used to build training trajectories (Sec 3.1 Step 0 partitions the pool; Sec 3.4 samples from it), and no overlap or contamination analysis is reported. If test videos or questions overlap with the 30K VQA pairs / 7K trajectories, the reported improvement over Claude-4.5-Sonnet could reflect memorization of the training distribution rather than tool-augmented exploration. The unresolved 100-vs-200 benchmark-size inconsistency compounds this: on 100 items a 5-point gap is about 5 questions, well within chance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Video-DeepResearch (Video-DR), a framework for multimodal deep-research agents that operate over continuous video streams rather than static images. The method combines a decoupled perception-exploration pipeline (Select_Keyframe and Crop_Search before text/web search) with a two-stage training recipe: SFT on 7K synthesized video-grounded trajectories plus 7K text-only QA instances, followed by GRPO on a 2K moderate-difficulty RL set. The authors also introduce VIDEODR-BENCH, a human-AI collaboratively annotated benchmark of multi-hop video VQA instances, and claim their Video-DeepResearch-35B-A3B reaches 64.0% average accuracy, surpassing Claude-4.5-Sonnet (59.0%) by 5.0 points. The paper reports substantial gains over the base models, an ablation study, and a tool-usage analysis showing reduced modality bias.","tokens_in":19378,"tokens_out":3583,"duration_ms":40135,"significance":"If the central claims hold, the paper makes a meaningful contribution: it identifies two concrete failure modes of current multimodal agents on video deep research (visual tool aversion and parametric knowledge leakage), and it provides a reproducible training recipe that improves compact open models beyond far larger proprietary systems. The human-AI collaborative benchmark construction and the public code release are also valuable assets to the community. However, the significance is conditional on three load-bearing issues: an unresolved inconsistency in benchmark size, insufficient evidence for the claimed leakage-free and tool-requiring properties of the benchmark, and a circular judge model shared between RL training and evaluation. These issues directly affect whether the headline 5.0-point SOTA improvement is meaningful.","major_comments":[{"comment":"The benchmark size is internally inconsistent. The abstract and conclusion state VIDEODR-BENCH comprises 200 instances, while Section 4.1 explicitly says it 'comprises 100 human-annotated VQA pairs.' Table 3's category percentages (29.5 + 22.0 + 18.5 + 14.5 + 12.0 + 3.5 = 100.0) are consistent with 100, not 200. Because the headline 5.0-point SOTA margin is computed on this benchmark, a 100-instance test makes the margin equivalent to roughly 5 questions. Please state the exact number, report per-category counts, and provide confidence intervals or a significance test for the claimed differences.","section":"Abstract / Section 4.1 / Section 6 / Table 3"},{"comment":"The claim that 'every question provably requires both visual search and external knowledge reasoning' is not supported by the described filtering. Section 3.1 Step 2 discards a QA pair only if Qwen3.5-397B-A17B answers correctly in any of four tool-free rollouts; Section 3.4 applies a similar filter. This only shows that one particular model could not answer from memory; it says nothing about GPT-5, Claude-4.5-Sonnet, or the trained models, and it does not establish that both visual search and external knowledge are necessary. Additionally, the benchmark is sampled from the same curated video pool used to construct the 30K VQA / 7K training trajectories, and no overlap or contamination analysis is reported. If test items overlap with training data, the reported SOTA gap could reflect memorization rather than tool-augmented exploration. Please provide a quantitative contamination analysis","section":"Section 3.1 Step 2 / Section 3.4"},{"comment":"The evaluation loop is partially circular. Correctness on VIDEODR-BENCH is judged by Qwen3-VL-30B-A3B-Instruct (Section 4.1), and the same model is used to assign sparse RL rewards in GRPO (Section 3.3). Moreover, Video-DeepResearch-30B-A3B is fine-tuned from exactly Qwen3-VL-30B-A3B-Instruct. This shared judge can systematically favor the trained model's answer distribution. Please evaluate with an independent judge model or human evaluation, and report agreement between judges; at minimum, report results with a different, non-overlapping judge.","section":"Section 3.3 / Section 4.1"},{"comment":"The benchmark's central property that every item requires both visual search and external knowledge is not validated by any explicit analysis. The leakage filter only checks tool-free answerability; it does not verify that a question is unanswerable with text-only search or with visual-only reasoning. Given the paper's emphasis on modality bias, it is important to show, at least on a sample, that successful answers require both the visual grounding step and the web retrieval step. A decomposition experiment (e.g., text-only access, visual-only access, and full tool set) would substantiate the claim.","section":"Section 3.1 / Section 3.4 / Figure 2"}],"minor_comments":[{"comment":"The model name is inconsistent: Table 3 and Table 4 list 'Qwen3.5-397B-A13B,' while Sections 2, 3.1, and 3.2 refer to 'Qwen3.5-397B-A17B.' Please unify the nomenclature.","section":"Section 4.1 / Table 3 / Table 4"},{"comment":"The appendix title and text refer to 'VIDEOHUNT,' which appears to be a leftover name; it should be VIDEODR-BENCH throughout.","section":"Appendix D"},{"comment":"There is an unresolved reference: 'to address the under-utilization of text tools observed in Table??.' Please replace with the correct table number.","section":"Section 3.3"},{"comment":"The coordinate scale for select_crop_search is inconsistent: Table 6 specifies normalized [0,1] coordinates, while Figure 8 and the prompt use a 0–1000 scale. Please standardize and make the prompt consistent with the tool specification.","section":"Table 6 / Figure 8"},{"comment":"The ablation rows are labeled '4k-SFT' and '14k-SFT+2K-RL,' but the text describes '7K-SFT' and '2K-RL'; the relationship between the 4K and 14K labels and the 7K trajectory count should be clarified.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The core idea and the engineering effort are solid, but the submission currently cannot support its strongest claims. The 100-vs-200 benchmark-size inconsistency and the shared judge between RL and evaluation are the most urgent issues; both are fixable with additional experiments and analyses. If the authors provide a contamination analysis, an independent judge evaluation, and a corrected benchmark description, the paper could become a valuable contribution to the multimodal agent literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is worth taking seriously, but the central number should not be trusted until three problems are fixed. It is a real attempt to move deep-research agents from static images to continuous video, and the training recipe is coherent. But the benchmark has an unresolved size inconsistency, the judge is partially the same model family being trained, and the 'provably requires tools' guarantee is not actually a proof.\n\nWhat is new: the paper identifies two concrete failure modes with numbers—visual tool aversion and parametric knowledge leakage—and builds a decoupled perception-exploration pipeline with stage-wise tool unlocking, trained with SFT plus GRPO. The ablation is sensible: each stage adds a measurable gain, and the tool-usage shift in the trained models is the strongest empirical point. The related work is adequate, including recent image-based deep research and video tool-calling systems. If the code and benchmark are actually released, this becomes a useful base for agentic video understanding.\n\nSoft spots, in order. First, the benchmark size: abstract and conclusion say 200 instances, Section 4.1 says 100, and the category percentages in Table 3 sum to roughly 100. On 100 items, a 5-point gap is five questions. They need to state the true number and give error bars or per-item annotations. Second, the correctness judge for both RL rewards and final evaluation is Qwen3-VL-30B-A3B-Instruct, and the 30B model is fine-tuned from the same model family. That is a real circularity; an independent or human judge is needed. Third, the leakage filter is not a proof: four tool-free rollouts of one model only show that one model could not answer correctly. Since the benchmark is sampled from the same video pool used to build training trajectories, the authors also need an overlap/contamination analysis. The stress-test note is on target here.\n\nWhat holds up: the failure-mode analysis and the pipeline design are plausible, and the paper is honest in its Limitations appendix about compute and human annotation costs. The core idea—force visual grounding before web exploration—is worth refereeing even if this version under-delivers on evaluation rigor.\n\nRecommendation: send to serious peer review. The task framing and training results are important enough to warrant referee time, but the reviewers should require a clean benchmark size, variance estimates, a more independent judge, and a contamination check before the 64.0% claim can be accepted at face value.","headline":"A genuinely new task framing and a sensible training recipe, but the headline SOTA gap rests on a benchmark whose size, judge, and leakage controls are not yet credible.","tokens_in":19973,"tokens_out":2663,"would_cite":false,"duration_ms":31031,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a video deep-research agent can beat proprietary models by forcing visual grounding before web retrieval, and backs the claim with a leakage-filtered benchmark.","keywords":["video deep research","multimodal agents","visual grounding","tool-augmented reasoning","parametric knowledge leakage","group relative policy optimization","multi-hop video QA","open-web exploration"],"falsifier":"First, resolve the printed inconsistency in benchmark size (200 in the abstract and conclusion, 100 in Section 3.4). Then take the actual benchmark, disable all tools, and let a strong model answer each question many times with varied decoding; if any non-trivial fraction is correct without tools, the 'provably requires both visual search and external knowledge' claim is falsified. A companion check is to search the benchmark videos' frames and source URLs in the SFT/GRPO training corpora for contamination.","tokens_in":18945,"feed_emoji":"🎬","tokens_out":11176,"duration_ms":113613,"temperature":0.7,"pith_summary":"The paper tries to establish that multimodal deep research can move from static images to continuous video, and that the right training pipeline—not raw model size—is what unlocks it. It argues that current agents fail this task for two reasons: they avoid visual tools in favor of text search (modality bias), and they answer from memorized knowledge instead of actually grounding in the video (parametric knowledge leakage). To counter both, the paper builds Video-DeepResearch: a decoupled perception-then-exploration pipeline that forces the agent to select keyframes and crop-search visual entities before any web text retrieval, trained with supervised fine-tuning followed by GRPO reinforcement learning. On a new human-AI collaborative benchmark of 200 multi-hop video questions that the paper claims provably require both visual search and external knowledge, the 35B variant reaches 64.0% average accuracy, beating Claude-4.5-Sonnet (59.0%), Gemini 2.5 Pro (57.5%), and GPT-5 (52.5%). If these results hold, they would show that agentic tool use for video can be taught at compact scale, and that existing benchmarks were partly measuring memorization rather than real video understanding.","feed_headline":"Video agent tops Claude-4.5, GPT-5 by grounding before searching","feed_subtitle":"A 35B open model hits 64.0% on a benchmark that forces real visual grounding, not memory recall.","key_machinery":"The mechanism that carries the argument is the decoupled perception-exploration pipeline with stage-wise tool unlocking. The agent is first restricted to two visual tools—Select_Keyframe, which isolates an informative frame from the video stream, and Crop_Search, which crops a bounding box around a salient entity and runs a reverse-image web search—and is forced to exhaustively ground visual entities across frames before the action space expands to Text_Search and Visit. This ordering compels the agent to build concrete visual evidence before web retrieval. The two-stage training recipe then locks in the behavior: supervised fine-tuning on 7K curated trajectories plus 7K text-only QA instanc","core_discovery":"The central discovery is that video deep research can be operationalized as sequential grounding: first find the informative frames, then crop the salient entities and run visual search on them, and only then turn to text web search. The paper identifies two failures in current agents—visual-tool aversion and parametric-knowledge leakage—and shows that a decoupled perception-exploration pipeline with stage-wise tool unlocking, plus SFT and GRPO training, corrects both. On the paper's Video-DR-Bench, Video-DeepResearch-35B-A3B reaches 64.0% average accuracy, surpassing Claude-4.5-Sonnet (59.0%) by 5.0 points and clearly outperforming GPT-5 (52.5%) and Gemini 2.5 Pro (57.5%); the 30B-A3B varia","pith_inferences":["If the leakage filter is airtight, the same tool-free rollout check could become a standard sanity test for any benchmark claiming to measure tool-augmented reasoning, not just for video.","The stage-wise tool unlocking principle suggests a transferable curriculum for multimodal agents: force competence in one modality's grounding before opening another modality's tools, which could extend to audio, egocentric, or live-streaming research agents.","The paper's claim that training methodology outweighs scale is conditional on a single 200-question benchmark; a direct test would be to scale the same pipeline to the 397B base model and evaluate on a larger, independent video-research set.","The weaker News-category gain for the 35B variant (8.4 points) compared with the 30B variant (16.6 points) hints that temporally dynamic content may need a dedicated temporal tool, which the paper does not develop."],"forward_implications":["Compact open models trained this way can outperform much larger proprietary models on this benchmark, so model scale alone is not enough for video deep research.","Benchmarks that do not filter parametric memory are partly measuring memorization; adding tool-use tracking and leakage filtering changes the rankings and the conclusions.","The pipeline's stages each contribute: the base 30B model scores 40.5%, trajectory SFT raises it to 53.0%, adding text QA SFT brings 56.8%, and GRPO yields 59.3%.","The training recipe changes agent behavior, not just accuracy: visual tool invocations rise from roughly 0.10 per task in the 397B baseline to 2.33 in the trained 30B variant, indicating a shift toward balanced multimodal exploration.","The RL stage adds a further gain over SFT alone, suggesting that autonomous exploration can be taught beyond the imitation ceiling."],"supporting_citations":[{"why":"Supplies the existing VideoDR benchmark used in the preliminary failure-mode study and in the main evaluation; its tool-call counts expose visual-tool aversion.","marker":"Liu et al., 2026"},{"why":"The prior image-centric Vision-DeepResearch system this work extends; also supplies the 7K text-only QA instances added during SFT.","marker":"Huang et al., 2026"},{"why":"Provides the Qwen3.5 foundation models—the 35B-A3B base for the larger variant and the 397B generator used for keyframe, entity, and trajectory synthesis.","marker":"Team, 2026"},{"why":"Provides the Qwen3-VL-30B-A3B-Instruct base for the 30B variant and the reward/judge model used in GRPO and answer grading.","marker":"Bai et al., 2025"},{"why":"Defines the GRPO objective that drives the reinforcement-learning stage beyond SFT imitation.","marker":"Shao et al., 2024"},{"why":"Claude-4.5-Sonnet is the proprietary baseline the 35B variant claims to surpass by 5.0 points.","marker":"Anthropic, 2025"},{"why":"GPT-5 is the proprietary baseline whose near-zero tool calls with competitive accuracy demonstrate parametric knowledge leakage.","marker":"OpenAI., 2025"},{"why":"Gemini 2.5 Pro is the third proprietary baseline against which the trained agents are compared.","marker":"Comanici et al., 2025"},{"why":"CLIP inter-frame similarity drives the keyframe-selection step in the data engine and evaluation preprocessing.","marker":"Radford et al., 2021"}],"fun_headline_variants":["Video agent that grounds before searching tops GPT-5, Claude-4.5","Decoupled grounding lifts video agent past GPT-5 and Claude","35B model beats GPT-5 by 11.5 points on video bench","Stage-wise tool unlocking propels video agent to new SOTA","Open video agent tops GPT-5, Claude-4.5 by grounding first"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The central claim stands on the assumption that no Video-DR-Bench question can be answered from the model's memorized knowledge—the paper filters with four tool-free rollouts plus human verification, and the whole 'provably requires tools' guarantee depends on that filter being airtight.","fun_headline_variants_meta":{"raw":{"variants":["Video agent that grounds before searching tops GPT-5, Claude-4.5","Decoupled grounding lifts video agent past GPT-5 and Claude","35B model beats GPT-5 by 11.5 points on video bench","Stage-wise tool unlocking propels video agent to new SOTA","Open video agent tops GPT-5, Claude-4.5 by grounding first"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000898,"raw_usage":{"total_tokens":3759,"prompt_tokens":850,"completion_tokens":2909,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":2811}},"tokens_in":594,"tokens_out":2909,"duration_ms":23478,"temperature":1.0,"reasoning_tokens":2811,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T04:38:36.360211+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"First, resolve the printed inconsistency in benchmark size (200 in the abstract and conclusion, 100 in Section 3.4). Then take the actual benchmark, disable all tools, and let a strong model answer each question many times with varied decoding; if any non-trivial fraction is correct without tools, the 'provably requires both visual search and external knowledge' claim is falsified. A companion check is to search the benchmark videos' frames and source URLs in the SFT/GRPO training corpora for contamination.","supporting_citations":[],"review_version":1}