{"id":"bd1e15d5-ca83-426a-bc20-452cb1da5564","arxiv_id":"2508.17932","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"CAVIA, a training-free loop where an LLM directs a VLM to inspect specific video frames and repeats until confident, reports accuracy gains on EgoSchema, NExT-QA, and IntentQA, but those gains are not shown to come from the loop itself.","lead":"A video question answering system that makes a computer watch a video, ask itself what information it is missing, look again at specific frames, and repeat until confident, reports accuracy gains on three long-video benchmarks. The method needs no training, but its reported wins come from comparisons across different AI models, not controlled tests.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is not causally supported: abstract mixes GPT-4 and GPT-4.1 results, and no ablation isolates the closed-loop mechanism from backbone strength or compute budget.","rationale":"The reader's weakest assumption identifies the backbone confound, and the paper's own text confirms it: the abstract selects different GPT versions per benchmark, and the only direct same-backbone comparison (CAVIA GPT-4 vs VideoTree GPT-4) is not accompanied by a compute-matched or prompting-matched ablation. My stress-test broadens this slightly to include the missing open-loop control and the empty Limitations section, but the reader's core objection stands. Good faith requires acknowledging that CAVIA's GPT-4 row does beat VideoTree's GPT-4 row on EgoSchema Full and IntentQA, so the architecture may have some effect; however, the paper's central claim is specifically that the closed-loop coordination is what drives the gains, and that claim is never directly tested. Without a single-pass counterpart that uses the same extractor, the same prompts, and the same token budget, the reported improvements remain confounded with additional compute and backend capability. The verdict of REJECT is therefore appropriate, not because the idea is wrong, but because the evidence presented does not establish the causal mechanism claimed.","tokens_in":3646,"tokens_out":4187,"duration_ms":38857,"concrete_test":"Implement an open-loop control of CAVIA with identical modules and budget: use the same GPT-4 LLM and the same VLM, the same hierarchical caption-cluster localization, and the same final answer prompt, but remove only the iterative confidence-driven feedback—supply all selected captions (matched to the same number of captions at the closed-loop convergence) in one pass, and cap total LLM tokens to the closed-loop total. Evaluate on EgoSchema Full and NExT-QA. If the single-pass version matches CAVIA's accuracy within about 1%, the reasoning-perception loop is not the cause of the gains; if it drops materially, the loop is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the closed-loop reasoning-perception coordination, not the underlying models, produces the reported gains. Table 1 is the main evidence, but it compares CAVIA against systems with different backbones, prompt designs, and total LLM/VLM calls. The abstract selects GPT-4 for EgoSchema (65.7%) and GPT-4.1 for NExT-QA (76.1%) and IntentQA (73.8%), so the headline 'state-of-the-art' numbers are not achieved by a single configuration and have no same-backbone baseline at the GPT-4.1 level. The only same-backbone comparison is CAVIA GPT-4 vs VideoTree GPT-4, which shows gains on EgoSchema Full and IntentQA but only +1.0 on NExT-QA average; even these gains could be explained by the larger number of captions or longer iterative reasoning, since no compute-matched open-loop control is provided. The internal ablation (Fig. 3) contrasts text-only, video-only, and full CAVIA, but omits a single-pass open-loop variant using the same hierarchical localization and the same final prompt. Figure 4 shows accuracy increases with iterations, but does not compare against an equally expensive static pipeline. Additionally, the Limitations section is present but empty, and no code or prompt templates are released, preventing verification of the reported numbers. Thus the causal attribution to the loop is untested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CAVIA, a training-free closed-loop video question answering framework that iterates between an LLM reasoner and a VLM perceiver. It claims state-of-the-art results on EgoSchema, NExT-QA, and IntentQA through hierarchical reasoning-guided localization, cross-modal semantic bridging, and confidence-driven iterative synthesis. The manuscript includes comparisons with prior systems, ablations over modalities and refinement iterations, and an efficiency analysis.","tokens_in":3944,"tokens_out":6638,"duration_ms":61786,"significance":"If the closed-loop mechanism were shown to be the cause of the reported gains, the work would be a useful contribution to long-form video QA: it is training-free, integrates off-the-shelf models, and addresses a real limitation of static pipelines. The paper also makes an effort to study the effect of the number of refinement iterations and the number of captions. However, because the current experiments do not isolate the loop from backbone choice or compute budget, the central claim of a scalable paradigm is not yet substantiated.","major_comments":[{"comment":"The headline numbers in the abstract are not produced by a single system configuration: EgoSchema uses CAVIA-GPT-4 (65.7%), while NExT-QA (76.1%) and IntentQA (73.8%) use CAVIA-GPT-4.1. Prior SOTA baselines are GPT-3.5 or GPT-4, so the '+5.3%, +2.6%, +6.9%' gains are confounded by the choice of a stronger, never-compared backbone (GPT-4.1). The only same-backbone comparison, CAVIA-GPT-4 vs. VideoTree-GPT-4, shows smaller gains and a loss on Descriptive questions (79.2 vs. 81.3). The abstract should report a single configuration with matched baselines, and the paper should add GPT-4.1 baselines for the prior methods.","section":"Abstract and Table 1"},{"comment":"The ablation that is meant to validate the closed-loop design compares text-only, video-only, and the full CAVIA pipeline, but it does not include the crucial open-loop control: a single-pass variant that uses the same hierarchical localization and the same final reasoning prompt but skips the iterative feedback. Without this control, the observed gains could be due to the larger number of captions or the longer reasoning trace rather than to the feedback loop itself. The same concern applies to Figure 4, which plots accuracy versus refinement iterations but does not compare against an equally expensive static pipeline.","section":"§4.2.2 and Figure 3"},{"comment":"The manuscript's 'Limitations' section contains only the sentence fragment 'While CA VIA demonstrates strong performance across multiple benchmarks, several limitations' and then ends; it does not state any limitations. This is a missing statement of limitations, and it matters because the method has evident limitations (dependence on the backbone LLM, the number of iterations as a free parameter, API cost, and unknown generalization to other video domains). In addition, no code, prompts, or evaluation scripts are provided, so the reported numbers cannot be independently verified.","section":"Limitations section"},{"comment":"No standard deviations, confidence intervals, or significance tests are reported for any of the main results (Table 1, Figures 3-5). The trend line in Figure 4 is fitted to only four points (refinement iterations 1-4); the reported r=0.932 is not statistically meaningful for such a small sample and should not be used to claim a monotone 'consistent improvement pattern.'","section":"Table 1 and Figures 3-5"}],"minor_comments":[{"comment":"The method name is typeset inconsistently as 'CAVIA' in the abstract and 'CA VIA' throughout the body; please unify the notation.","section":"Throughout"},{"comment":"Table 1 would be more informative if it reported the number of LLM/VLM calls or captions used by each method, since the central comparison is about efficiency as well as accuracy.","section":"Table 1"},{"comment":"Figure 5 compares accuracy versus number of captions, but the figure caption does not specify whether the LLoVi and VideoTree points are taken from the original papers or re-run under the same conditions; this should be stated.","section":"Figure 5"},{"comment":"The abstract states 'state-of-the-art performance' with the three benchmark numbers, but because these numbers come from two different configurations, the claim should be qualified.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The core idea is plausible, but the experimental validation currently does not support the causal attribution to the closed-loop mechanism. The authors should be asked to provide same-backbone baselines and an open-loop control before the paper can be considered for publication. If they are unable to provide these controls, the paper should be rejected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is sound: instead of captioning the whole video once, let the LLM repeatedly ask the VLM for the specific frames and details it needs. That is a genuine extension of agentic video QA, and the combination of hierarchical localization, semantic bridging, and confidence-based stopping is new relative to DrVideo and VideoTree. The paper is clearly written and the system is described well enough to reconstruct. I believe the authors are onto something real.\n\nThe problem is the evidence. The abstract's state-of-the-art claim mixes GPT-4 and GPT-4.1 results, choosing whichever version gives the best number per benchmark. That alone would make me suspicious. The only same-backbone comparison is CAVIA GPT-4 vs VideoTree GPT-4, and while those gains are positive, the paper never shows that the closed loop is what causes them. The ablation in Figure 3 contrasts text-only, video-only, and the full system, but omits a single-pass open-loop variant using the same hierarchical localization and the same final prompt. Figure 4 shows accuracy rising with iterations, but no comparison against an equally expensive static pipeline. So the central causal claim—that the reasoning-perception loop drives the gains—is untested. The efficiency analysis (Figure 5) is a nice attempt, but it only measures captions, not total compute or latency.\n\nOther soft spots: the Limitations section is literally empty, terminating at \"several limitations\" with no content. That is unprofessional and, combined with no code or prompt templates, makes verification hard. The trend line in Figure 4 is fit to four points and reported with r=0.932, which is not meaningful without error bars or more samples.\n\nThe benchmark gains against prior SOTA are computed against different backbones and prompt designs, so they are not apples-to-apples. I do not think the paper is dishonest—the method is described transparently, and the related work is fair. But the headline overclaims what is actually supported.\n\nFor a reader: this is a useful example of a promising architecture whose evaluation falls short of its claims. With a compute-matched open-loop control and consistent backbone across comparisons, the paper could be solid. Without that, the numbers don't prove the loop matters. I would not cite the SOTA numbers, but I might cite the method in a related-work survey.\n\nRecommendation: send it to peer review, but with a clear request for a controlled ablation and de-cherry-picked abstract. A serious referee can make the paper address the confound. Desk-rejecting would waste a plausible idea; accepting as is would let a shaky causal claim into the literature.","headline":"Plausible method, unproven headline: the loop idea is sensible but the experiments never isolate it from backbone strength or compute budget.","tokens_in":4459,"tokens_out":2330,"would_cite":false,"duration_ms":23727,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A closed-loop reasoning-perception cycle, in which an LLM tells a VLM what to look at next, improves long-form video question answering accuracy by up to 6.9 points over static pipelines.","keywords":["video question answering","closed-loop reasoning","reasoning-perception coordination","iterative refinement","long-form video understanding","training-free","multimodal prompting","keyframe localization"],"falsifier":"Run the same LLM, same VLM, same prompt templates, and same caption budget with the loop disabled—so the VLM extracts in one fixed pass and the LLM answers from that static evidence—and compare accuracy on EgoSchema, NExT-QA, and IntentQA; if the static pass matches or beats CAVIA, the central claim is false.","tokens_in":3469,"feed_emoji":"🎬","tokens_out":6730,"duration_ms":62517,"temperature":0.7,"pith_summary":"Long-form video question answering usually runs perception first and reasoning second: a vision model absorbs the video into captions or frame features, and an LLM answers from that fixed summary. The paper argues this decoupling is the core failure mode, because different questions need different visual evidence from the same footage. It presents CAVIA, a training-free framework that closes the loop: the LLM states what information it lacks, the vision model extracts precisely that, and the cycle repeats until the LLM is confident. On EgoSchema, NExT-QA, and IntentQA the closed loop outperforms open-loop baselines by +5.3, +2.6, and +6.9 points, which the paper takes as evidence that reasoning-perception coordination is a scalable paradigm.","feed_headline":"Video QA gains up to 6.9 points when reasoning guides perception","feed_subtitle":"Training-free CAVIA repeatedly asks the vision model for missing evidence, beating static pipelines on three benchmarks.","key_machinery":"The load-bearing object is the closed-loop reasoning-perception cycle itself: an LLM reasons about the query, identifies a specific information gap, emits a targeted prompt (e.g., \"describe the tool manipulation sequence in frames 45–52\"), a VLM extracts that visual evidence, and the new information is folded back into the LLM's reasoning until a confidence metric signals convergence. The three named components—hierarchical reasoning-guided localization, cross-modal semantic bridging, and confidence-driven iterative synthesis—are the mechanisms that make this loop precise rather than a generic caption-correction pass.","core_discovery":"The paper's central claim is that a closed reasoning-perception loop—where an LLM's reasoning gaps determine what a VLM extracts next—improves long-form video understanding without any training. The discovery is operationalized in CAVIA's three components: hierarchical reasoning-guided localization narrows from caption clusters to precise frames; cross-modal semantic bridging turns each identified reasoning gap into a targeted spatial-temporal extraction prompt; confidence-driven iterative synthesis repeats this cycle until the answer stabilizes. The paper reports state-of-the-art results on EgoSchema (65.7%), NExT-QA (76.1%), and IntentQA (73.8%), and interprets the gains on temporal and causal question types as direct evidence that query-driven visual extraction recovers information that static pipelines miss.","pith_inferences":["An implication the paper leaves implicit is that the architecture's benefit can only be isolated if the same LLM, VLM, prompt templates, and compute budget are run with the loop open and closed; without that controlled experiment, the reported gains may overstate the architecture's contribution.","The excerpt stops at the opening line of the paper's own Limitations section (“While CAVIA demonstrates strong performance... several limitations”); reading the full list would clarify what the authors themselves consider unresolved, and that list should be weighed before accepting the paradigm claim.","A testable extension beyond the paper: measure whether the number of refinement iterations predicts answer correctness per question type; if it does, the confidence score could serve as a reliability oracle for deferring hard questions to a stronger model."],"forward_implications":["Systems that currently caption the whole video first can be restructured so that the query drives what is looked at, without retraining the underlying LLM or VLM.","Iterative refinement with targeted prompts should be expected to produce the largest gains on temporal and causal question types, where the needed evidence is dispersed across frames.","The confidence-driven stopping rule gives a free efficiency knob: a deployment can trade compute for accuracy, since the loop halts once the answer is judged stable.","The same closed-loop design should transfer to other multimodal tasks that pair a reasoner with a perception module, such as temporal grounding or embodied decision-making, where evidence needs vary by query."],"supporting_citations":[{"why":"Provides the SeViLA vision-centric keyframe localization baseline that CAVIA must exceed on NExT-QA and IntentQA.","marker":"(Yu et al., 2023b)"},{"why":"Provides the VideoTree GPT-4 baseline, the strongest open-loop hierarchical competitor on EgoSchema and NExT-QA.","marker":"(Wang et al., 2025c)"},{"why":"Provides the LLoVi GPT-4 open-loop captioning pipeline baseline on all three benchmarks.","marker":"(Zhang et al., 2024a)"},{"why":"Provides the VideoAgent iterative-refinement baseline that lacks fine-grained temporal localization and targeted prompting.","marker":"(Wang et al., 2025a)"},{"why":"Represents the prior iterative caption-correction approach that CAVIA contrasts with, showing simple correction is insufficient.","marker":"(Zhang et al., 2025)"}],"fun_headline_variants":["Training-free loop lifts video QA by up to 6.9 points","Reasoning guides vision to improve video QA without training","Closed-loop perception boosts video QA, no training needed","CAVIA: reasoning-driven visual loop raises video QA scores","Video QA gains from query-aware reasoning-guided perception"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported gains come from the closed-loop architecture rather than from the choice of LLM backbone, prompting, or compute; the manuscript's own Limitations section (which begins after the conclusion but is cut off in this excerpt) acknowledges unresolved issues, and the main comparison table mixes GPT-4 and GPT-4.1 results, selecting the best per benchmark in the abstract.","fun_headline_variants_meta":{"raw":{"variants":["Training-free loop lifts video QA by up to 6.9 points","Reasoning guides vision to improve video QA without training","Closed-loop perception boosts video QA, no training needed","CAVIA: reasoning-driven visual loop raises video QA scores","Video QA gains from query-aware reasoning-guided perception"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000956,"raw_usage":{"total_tokens":4058,"prompt_tokens":914,"completion_tokens":3144,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":3064}},"tokens_in":530,"tokens_out":3144,"duration_ms":25430,"temperature":1.0,"reasoning_tokens":3064,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:58:19.043133+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same LLM, same VLM, same prompt templates, and same caption budget with the loop disabled—so the VLM extracts in one fixed pass and the LLM answers from that static evidence—and compare accuracy on EgoSchema, NExT-QA, and IntentQA; if the static pass matches or beats CAVIA, the central claim is false.","supporting_citations":[],"review_version":2}