{"id":"624875de-e807-4241-8d60-01b3c05fd302","arxiv_id":"2509.07680","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CAViAR, an agent-plus-critic system for long video reasoning, improves on direct video LLM inference across LVBench, Neptune, and ActivityNet-RTL.","lead":"The paper pairs an LLM agent that writes and executes video-analysis programs with a critic that picks the best of several reasoning strategies. The combination beats prior methods on long-video QA and reasoning temporal localization benchmarks without any training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Critic few-shot examples may be drawn from the test distribution; if so, the LVBench SOTA gain is not an independent measure.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: the critic's few-shot examples may come from the evaluation distribution, and the reported gains are largely due to the critic. My read of the full text and supplement confirms this is the most fragile point. The agentic framework is plausible, the ablations are informative, and the module API is clearly described, but the 62.0 vs 48.7 LVBench claim would not be an independent measure if the critic's examples are test-derived. The paper's own statement that the examples are 'written by the authors' with the aim of being 'in-distribution' is insufficient disclosure: it does not rule out authorship after looking at the test set, especially since LVBench has no training split. The concrete check of matching examples to evaluation items would settle whether the concern lands; if it does not land, the conditional verdict can be upgraded. The existing CONDITIONAL verdict already reflects this uncertainty, so I recommend no change.","tokens_in":49403,"tokens_out":4322,"duration_ms":55180,"concrete_test":"Extract the four critic examples per dataset from Supplement Listings 4-6 and compare them against the corresponding evaluation sets by exact query text, answer choices, and video identifiers, plus near-duplicate matching on paraphrased queries. If any critic example overlaps with an evaluation instance, the headline result is invalidated by test-set leakage. If no overlap is found, also rerun LVBench with the critic given zero in-context examples; if accuracy collapses toward the agent-only 47.1, the method's reported gain is fragile and should be reported as a conditional, example-dependent result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result depends almost entirely on the critic: on LVBench, the agent alone scores 47.1 and the agent+critic scores 62.0 (Table 4). The critic's only supervision is four in-context examples per dataset, described in Supplement Section 4 as 'written by the authors with the aim of being in-distribution for the datasets shown.' The printed examples are not generic templates: they use LVBench-style questions, exact multiple-choice answer formats, and specific celebrity/vlogger videos (e.g., Jimmy Kimmel's suit color, Harry Styles 'photo of six people', a vlogger buying makeup); the ActivityNet-RTL examples in Listing 6 mirror the evaluation queries. If any of these examples were drawn from the evaluation sets, or authored after inspecting them, the critic has effectively been fitted to the test distribution. Because LVBench has no training/validation split (Section 4.1: 'No supervised methods are available to compare to as there is no training set'), there is no legitimate development set from which to select these examples. The manuscript does not disclose the provenance of the critic examples, so the central SOTA claim cannot be independently verified as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CAViAR, an inference-time framework for long-video reasoning that pairs a program-generating LLM agent with a natural-language critic. The agent iteratively composes executable programs from a small set of video modules (retrieval-QA, temporal grounding, temporal localization, ASR understanding), while the critic selects among several sampled reasoning traces produced from different module subsets. Experiments are reported on LVBench, Neptune, and ActivityNet-RTL, with additional results on EgoSchema and GPT-4o-mini in the supplement. The main claims are that CAViAR improves over direct inference with the same base model on all three tasks and achieves a 62.0% accuracy on LVBench, a 13.0-point absolute gain over the reported state of the art, and 32.3 mIOU on ActivityNet-RTL. The ablations show that the critic is the main driver of the LVBench gain (47.1 without critic vs 62.0 with critic) and that fixed single-program generation is far weaker.","tokens_in":49668,"tokens_out":3401,"duration_ms":42669,"significance":"If the reported numbers are reliable and independent, the paper makes a useful contribution: it demonstrates that iterative, adaptive tool use plus a lightweight critic can outperform much larger or specially trained video models on long-video reasoning, without additional training. The method is interpretable, and the ablations are well designed: comparing against single-program baselines, an oracle fixed-module selection, and a confidence self-evaluation baseline gives the reader a clear picture of where the gains come from. The supplement provides full prompts, module APIs, and critic examples, which is a genuine strength. The central concern is that the critic's only supervision is four in-context examples per dataset, whose provenance is not disclosed; because LVBench has no training/validation split, the examples may have been selected from or authored after inspecting the evaluation distribution. Since the critic is responsible for most of the reported improvement, the headline SOTA claim is not independently verifiable as written. The paper also does not provide code or error bars, further limiting verification.","major_comments":[{"comment":"The critic's only supervision is four in-context examples per dataset, and the supplement states these are 'written by the authors with the aim of being in-distribution for the datasets shown.' The printed examples are not generic templates: they contain concrete LVBench-style queries (e.g., Jimmy Kimmel's suit color at 12:10, Harry Styles and a photo of six people, a vlogger buying makeup at Kicks), and §4.1 states LVBench has no training/validation split. If any of these examples were drawn from the evaluation set, or authored after inspecting it, the critic is effectively fitted to the test distribution, and the 62.0 vs 47.1 critic gain in Table 4 is not an independent measure of the method. The manuscript must disclose the exact source of every critic example, should rule out test-set overlap, and ideally should re-run with a development set or with examples that are clearly not draw","section":"Supplement §4, Listing 4; §4.1"},{"comment":"The section labelled 'EgoSchema critic examples' repeats the LVBench in-context examples verbatim, including the Jimmy Kimmel suit-color and vlogger/makeup items. If this is not a formatting artifact of the supplement, then the EgoSchema critic was given LVBench examples, contradicting the claim that the examples are 'in-distribution for the datasets shown.' This also reinforces the need for full provenance disclosure of critic examples for every dataset.","section":"Supplement Listing 7 (EgoSchema critic examples)"},{"comment":"All headline results are single point estimates. The checklist justifies the absence of error bars by stating that experiments are 'run in a deterministic setting,' but the method invokes LLMs with sampling and multiple strategy traces, and no code is released to reproduce the runs. Given that the central claim is a 15-point gap on LVBench and a 9-point mIOU gain, the paper should report variance across repeated runs (or provide code and exact API configuration so the runs can be independently reproduced). Without this, it is impossible to assess whether the gains are stable or partly due to a favorable random seed / example choice.","section":"§5, Table 4; NeurIPS Checklist §7"},{"comment":"The strategy subsets given to the reasoning agent are a per-dataset design choice that is not specified a priori. The oracle 'Optimal Fixed Modules' row shows that much of the critic's gain can be replicated by choosing the right fixed module subset (59.8 vs 62.0 on LVBench; 76.5 vs 77.2 on Neptune). The paper does not report how the three subsets were selected, or whether this selection was made before or after seeing evaluation results. Since the contribution claims to 'avoid dataset-specific tuning,' the subset selection procedure should be stated in the method and, ideally, validated on a held-out development set.","section":"§3.4, §5, Table 4"}],"minor_comments":[{"comment":"Typo: 'in a a variety of domains' should read 'in a variety of domains.'","section":"§1, Introduction"},{"comment":"The critique text says 'Strategy B fails as the retrieval_qa tool alone cannot actually locate a given timestamp in a video,' but Strategy B is labelled 'direct retrieval_qa' and does not explicitly call retrieval_qa as a tool. Clarify the intended strategy description so readers can follow why the tool cannot localize timestamps.","section":"Supplement Listing 4, Strategy B critique"},{"comment":"The caption states items in italics require oracle validation accuracy, but the difference between 'Single Program (Optimal Fixed Modules)' and 'Agent (Optimal Fixed Modules)' is not explained in the main text. A sentence defining how the optimal fixed subset is found would improve interpretability.","section":"Table 4 caption"},{"comment":"The GPT-4o-mini result is reported on 25% of LVBench without specifying how the 25% subset was sampled. Please state the sampling method and, if possible, whether the same subset is used for direct inference and CAViAR.","section":"Supplement §5.2"}],"recommendation":"major_revision","confidential_remarks":"The central methodological concern is the undisclosed provenance of the critic's in-context examples. The paper's own checklist states that all settings and prompts are provided in full, but the source of the critic examples is not disclosed, and the examples resemble evaluation queries. Given that the critic is the main driver of the reported gains, the editor should treat this as a correctness-risk issue rather than a mere reproducibility matter. In addition, the paper reports no code and no error bars, which, together with the provenance issue, makes the headline SOTA claim currently unverifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the bottom line: this is a sensible agent-plus-critic recipe for long-video reasoning, and the ablations are genuinely informative. The agent iteratively composes programs over video modules, and a critic picks among strategies defined by different module subsets. The gains on LVBench, Neptune, and ActivityNet-RTL are large and consistent—LVBench 62.0 vs 46.0 direct, and the critic is the main driver (47.1 agent vs 62.0 with critic). That is a real result if the evaluation is clean.\n\nWhat is new: the particular combination of an adaptive program-writing agent with strategy sampling and a natural-language critic that selects among sampled module subsets. The parts are not new—iterative tool use and verifier critics exist in text and image domains—but the video-specific module set and the strategy-sampling procedure are a legitimate new application. The method requires no training and produces interpretable traces, which is worth something.\n\nWhere it gets soft: the critic's four in-context examples per dataset. The supplement says they were 'written by the authors with the aim of being in-distribution for the datasets shown,' but the printed examples look like actual benchmark items: a Jimmy Kimmel suit-color question in LVBench format, a Harry Styles 'photo of six people' question, a vlogger buying makeup, and ActivityNet-RTL queries that mirror the evaluation set. LVBench has no training/validation split, so there is no legitimate development set to draw from. If any of these examples come from the evaluation sets, the critic has been fitted to the test distribution in the very component responsible for the headline jump. That would not sink the whole approach, but it would mean the reported SOTA numbers are not an independent measure of the method as described. The paper should disclose the provenance, rerun with examples held out from the test sets, and report variance; 'deterministic setting' in the checklist does not hold for LLM sampling.\n\nAlso, no code, no error bars, and the claims about avoiding dataset-specific tuning are undercut by the dataset-specific critic examples. The ablations with oracle fixed modules are a good idea and honestly labeled.\n\nBottom line: the method is plausible and the writing is clear. This deserves a serious referee, but the referee should push hard on critic-example provenance and demand a held-out evaluation. I would not cite the headline numbers until that is resolved.","headline":"A plausible agent+critic recipe for long-video reasoning with strong numbers, but the critic's few-shot examples look like they could come from the test sets, and until that is disclosed the headline gains are not independent.","tokens_in":50181,"tokens_out":2351,"would_cite":false,"duration_ms":27204,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CAViAR claims that pairing an iterative program-writing agent with a critic that selects among sampled video-reasoning strategies lifts accuracy on long-video benchmarks, reaching 62.0 percent on LVBench versus a prior best of 48.7.","keywords":["video reasoning","long-video question answering","agentic reasoning","tool use","reasoning critic","temporal localization","multimodal large language models","inference-time scaling"],"falsifier":"Run CAViAR on LVBench with critic examples provably held out from the evaluation set—written from a different set of long videos—and check whether the 62.0 percent accuracy and the 13-point gap over the prior best persist. As a direct check, determine whether the critic's staged 'host suit color at 12:10' example appears in the LVBench evaluation set; if it does, the critic has been calibrated on test-like items.","tokens_in":49322,"feed_emoji":"🎬","tokens_out":4889,"duration_ms":56465,"temperature":0.7,"pith_summary":"CAViAR is a method for complex long-video reasoning that adds no training: a language-model agent writes and runs short Python programs that call video modules such as frame retrieval, temporal grounding, localization, and ASR understanding, reading each tool's output before choosing the next step. Because a single run can fail, the method samples several module subsets to produce distinct reasoning traces, then a natural-language critic compares the traces and selects the most promising one. The paper reports 62.0 percent accuracy on LVBench versus 48.7 for the prior best, gains over direct inference on Neptune and ActivityNet-RTL, and ablations showing the critic is responsible for most of the improvement. If these numbers hold, existing perception models can be assembled at inference time to outperform much larger or specially trained systems on reasoning-heavy video tasks.","feed_headline":"Video agent plus critic hits 62% on LVBench, a 13-point jump","feed_subtitle":"A language-model critic chooses among sampled reasoning traces, lifting long-video QA without retraining the model.","key_machinery":"The central mechanism is a reasoning trace: a full sequence of programs and tool results produced by an agent that can inspect each result before writing the next program. The second component is the reasoning critic, a same-base-model natural-language ranker that is shown several sampled traces at once and, using a handful of in-context examples per dataset, names which strategies are likely to win. The video modules—retrieval_qa, get_segment, find_when, asr_understanding, think, and finish—are the tools the agent composes.","core_discovery":"On its own terms, the work claims that adaptive, verifiable reasoning beats fixed procedures for long-video understanding: a reasoning agent that iteratively composes video modules into executable programs, observing each result before deciding the next step, outperforms single-shot program generation, and adding a critic that chooses among multiple such reasoning traces yields the largest gains. The key empirical evidence is a 13-point absolute improvement on LVBench over the prior state of the art (62.0 vs 48.7), plus improvements over direct inference on Neptune (77.2 vs 74.9 with ASR manually provided) and on ActivityNet-RTL temporal localization (32.3 vs 23.0 mIOU). Ablations show the a","pith_inferences":["If the critic's in-context examples are genuinely held out from the evaluation sets, CAViAR's procedure should transfer to new datasets with only a handful of hand-written examples; a direct test is to swap the critic examples for ones written from a separate pool of long videos and measure the accuracy change.","The same generate-and-critique loop could apply to other settings where tool outputs are noisy and strategies diverge, such as embodied instruction following or document question answering, because the critic only reads text traces and never re-processes the video.","The gap between agent-plus-optimal-fixed-modules (59.8 on LVBench) and agent-plus-critic (62.0) suggests that sampling more than three strategies, or making strategies differ more sharply, could yield further gains—an extension the paper does not test.","One open question the paper's design raises, not settled by its experiments, is leakage: the critic's examples resemble actual benchmark items, so their provenance determines whether the reported gains measure generalization or test-set fitting."],"forward_implications":["The same base model can be scaled at inference time without any training: Gemini 1.5 Flash goes from 46.0 to 62.0 on LVBench and from 23.0 to 32.3 mIOU on ActivityNet-RTL under CAViAR.","Single-program approaches fail on these tasks: generating one fixed program yields near-random accuracy (27.1 on LVBench, 28.7 on Neptune), whereas the iterative agent recovers to 47.1 and 72.5 respectively.","The critic is the load-bearing addition: replacing it with confidence-based self-evaluation drops LVBench accuracy from 47.1 to 39.9, while the critic raises it to 62.0.","Critic-based strategy selection outperforms even the best fixed module subset found with oracle validation, suggesting the gain comes from per-query flexibility rather than from tuning module choice per dataset.","The approach is model-agnostic in principle: a preliminary experiment with GPT-4o-mini on 25% of LVBench still shows a substantial gain over direct inference."],"supporting_citations":[{"why":"Supplies the LVBench benchmark and the leaderboard baselines CAViAR compares against.","marker":"[29]"},{"why":"Supplies the Neptune dataset and the direct-inference-with-ASR baseline CAViAR improves on.","marker":"[20]"},{"why":"Introduces the ActivityNet-RTL reasoning temporal localization task and the LITA baseline.","marker":"[9]"},{"why":"Represents the single-program visual programming approach CAViAR contrasts with.","marker":"[6]"},{"why":"Defines API-based program generation with Python execution, the baseline style CAViAR extends to adaptive multi-step traces.","marker":"[23]"},{"why":"Provides the fixed three-stage modular video QA baseline that motivates adaptive reasoning.","marker":"[19]"},{"why":"Supplies the confidence-based self-evaluation baseline that the critic outperforms in ablations.","marker":"[30]"},{"why":"Motivates the critic by showing natural-language critiques can rank reasoning outputs better than scalar verifiers.","marker":"[37]"},{"why":"Supports the claim that single-program approaches are brittle and depend on hand-tuned APIs and examples.","marker":"[22]"},{"why":"Motivates presenting all strategies to the critic at once, based on the preference-between-options idea from RLHF.","marker":"[4]"}],"fun_headline_variants":["Critic-guided video agent tops LVBench by 13 points","Video agent with critic surpasses prior art on LVBench","Adaptive video reasoning: critic picks best trace, boosts LVBench","Agentic video reasoning with critic lifts LVBench to 62%","Critic-augmented agent achieves 62% LVBench, +13 over SOTA"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The critic's four in-context examples per dataset must teach it which module strategies succeed on the evaluation queries without being drawn from those same test sets; the supplement calls them author-written and in-distribution, but their content resembles actual benchmark questions, so leakage would invalidate the reported gains.","fun_headline_variants_meta":{"raw":{"variants":["Critic-guided video agent tops LVBench by 13 points","Video agent with critic surpasses prior art on LVBench","Adaptive video reasoning: critic picks best trace, boosts LVBench","Agentic video reasoning with critic lifts LVBench to 62%","Critic-augmented agent achieves 62% LVBench, +13 over SOTA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000409,"raw_usage":{"total_tokens":1938,"prompt_tokens":704,"completion_tokens":1234,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":1141}},"tokens_in":448,"tokens_out":1234,"duration_ms":11914,"temperature":1.0,"reasoning_tokens":1141,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T21:22:49.126950+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CAViAR on LVBench with critic examples provably held out from the evaluation set—written from a different set of long videos—and check whether the 62.0 percent accuracy and the 13-point gap over the prior best persist. As a direct check, determine whether the critic's staged 'host suit color at 12:10' example appears in the LVBench evaluation set; if it does, the critic has been calibrated on test-like items.","supporting_citations":[{"cited_title":"Morevqa: Exploring modular reasoning models for video question answering","cited_arxiv_id":null,"evidence_quote":"Defines API-based program generation with Python execution, the baseline style CAViAR extends to adaptive multi-step traces."},{"cited_title":"Analyzing Modular Approaches for Visual Question Decomposition","cited_arxiv_id":"2311.06411","evidence_quote":"Provides the fixed three-stage modular video QA baseline that motivates adaptive reasoning."},{"cited_title":"Eyes wide shut? exploring the visual shortcomings of multimodal llms","cited_arxiv_id":null,"evidence_quote":"Supplies the confidence-based self-evaluation baseline that the critic outperforms in ablations."},{"cited_title":"Self-chained image-language model for video localization and question answering","cited_arxiv_id":null,"evidence_quote":"Motivates presenting all strategies to the critic at once, based on the preference-between-options idea from RLHF."}],"review_version":1}