{"id":"41e2bc34-dc7e-4931-ad6e-4243601fba05","arxiv_id":"2505.08455","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new video benchmark asks models to reorder shuffled clips from everyday tasks; most large video language models perform near chance, and a recognize-then-reason prompt decomposition gives large gains.","lead":"This paper introduces VCRBench, 365 scrambled procedural videos that test whether video-language AI models can put the key steps of everyday tasks in the correct order. Most current models score at or below random guessing, while humans reach about 96%, showing a clear gap in causal reasoning from video.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Goal-name leakage weakens the visual-causal claim: the ordering signal is largely recoverable from the named goal plus language-level commonsense, so VCRBench may measure recognition plus language-based ordering rather than visual causal reasoning.","rationale":"The reader identified the same load-bearing assumption: the benchmark's validity as a test of video-based causal reasoning depends on the correct order not being inferable without genuine visual causal analysis. I agree, and the manuscript's own experiments strengthen the concern. The RRD decomposition separates recognition from reasoning, and the reasoning stage is purely textual: the prompt in Figure S2 gives the goal name and clip descriptions, and Table 3 shows that adding video back at that stage reduces accuracy. That is direct evidence that the causal-ordering signal used by the model resides in the language, not in the video. Consequently, the strongest claim—that LVLMs cannot model long-range causal dependencies from visual observations—is not uniquely supported: an alternative explanation is that they fail at open-ended visual event recognition/verbalization, or at integrating recognized events into a language-based causal chain, rather than failing at causal reasoning over visual input. This matters for the paper's central conclusion, not for the benchmark's basic utility: the human ceiling and consistent at-or-below-random results across many LVLMs are real and informative. I would not change the reader's conditional verdict. The condition is exactly that the linguistic-leakage interpretation be ruled out; the oracle-caption control and false-goal control would settle it. A secondary issue—the inconsistent RRD gains between Table 2 (49.9, +20.9) and Table 5 (54.2, +25.2) using different recognition settings without clear labeling—should also be fixed, but it is less central than the leakage question.","tokens_in":19751,"tokens_out":8540,"duration_ms":94243,"concrete_test":"Oracle-caption control on all 365 VCRBench samples: run the RRD causal-reasoning prompt (Figure S2) with the goal name, but replace model-generated clip descriptions with the human-annotated step names from CrossTask/WikiHow that were used to construct each video in Section 3.1. Feed this text-only input to a strong text LLM (e.g., GPT-4 or Gemini text API) and compute overall and step accuracy. If a text-only LLM with oracle descriptions approaches human-level accuracy (~96%), the causal-order information is fully contained in the goal name plus canonical step labels, confirming that VCRBench isolates recognition plus language ordering rather than visual causal reasoning. If this text-only accuracy stays near the 54.2% RRD level or lower, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"VCRBench's central diagnostic claim—that current LVLMs lack video-based long-form causal reasoning and cannot model long-range causal dependencies directly from visual observations—depends on the assumption that the shuffling task requires genuinely video-grounded causal inference. That assumption is not secure. The question template in Figure 4 explicitly names the goal ('Make Lemonade') and instructs the model to 'use your reasoning and common sense'; the RRD causal-reasoning prompt in Section 5.1 supplies the goal name and orders text descriptions; and Table 3 shows that reintroducing video at the reasoning stage degrades accuracy (49.9 to 46.6). Thus the causal-ordering component is largely solvable from the named goal plus procedural commonsense once each clip has been recognized and verbalized. Under this reading, VCRBench primarily tests (a) open-ended visual event recognition/verbalization and (b) language-based ordering, not causal reasoning performed over raw visual observations. The paper's own insight that LVLMs 'primarily rely on their language knowledge' supports exactly this ambiguity. If the order is recoverable from the goal plus clip descriptions, the benchmark cannot uniquely support the conclusion that LVLMs lack video-based causal reasoning. The main negative result remains useful as a multimodal-integration diagnostic, but the headline interpretation is not uniquely determined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"VCRBench is a new benchmark for long-form causal reasoning in video. The authors construct 365 test samples from CrossTask procedural videos and WikiHow step lists, shuffle 3-7 clips per task, and ask an LVLM to output the correct clip order. They evaluate 20+ open- and closed-source LVLMs, report that most open-source models perform at or below a random-guess baseline (e.g., Qwen2.5-VL-Instruct-72B: 29.0%) while Gemini-2.0-Flash-Thinking reaches 58.0% and humans reach 96.4%, and introduce Recognition-Reasoning Decomposition (RRD), a two-stage prompt (video recognition, then text-based ordering), which raises Qwen2.5-VL-72B to 54.2% under their best decomposition. They also report ablations showing that adding video to the reasoning stage degrades accuracy, and conclude that LVLMs mainly rely on language knowledge and cannot model long-range causal dependencies directly from visual observations.","tokens_in":19928,"tokens_out":7912,"duration_ms":82548,"significance":"VCRBench addresses a genuinely underexplored capability and its deterministic ordering format is a sensible middle ground between multiple-choice and open-ended QA. The paper's strengths are the breadth of the model suite, the high human accuracy (96.4%) indicating task clarity, the transparent construction from CrossTask/WikiHow, and the consistent direction of RRD gains across 7B-78B models. If the headline interpretation held, the finding that most LVLMs do not exceed random guessing would be an important diagnostic result. However, as discussed below, the goal-name leakage and the text-only reasoning stage mean the benchmark may be measuring event recognition plus language-level procedural ordering rather than causal reasoning over visual observations; the significance of the central claim therefore depends on the outcome of the proposed control experiments.","major_comments":[{"comment":"The central claim that VCRBench isolates video-based long-form causal reasoning is not yet established. The question template in Figure 4 explicitly names the goal (e.g., 'Make Lemonade') and instructs the model to 'use your reasoning and common sense'; the RRD causal-reasoning stage receives only the procedure name and text clip descriptions; and Table 3 shows that reintroducing video at that stage lowers accuracy (49.9 to 46.6). A model that can visually recognize and verbalize each clip and then order the resulting event descriptions using procedural commonsense can therefore score high without performing causal inference over raw visual input. To support the headline interpretation, the authors should add control conditions that remove or perturb the goal name (e.g., asking for the 'most plausible sequence' without naming the procedure, or using a wrong/absent goal), and compare RRD's text-only reasoning stage with an oracle that receives ground-truth step labels instead of model-generated descriptions. If a text-only oracle succeeds at ordering, the benchmark should be described as measuring event recognition plus language-level ordering, not visual causal reasoning per se.","section":"§3.1 (Figure 4), §5.1, Table 3"},{"comment":"The printed step-accuracy formula is missing the normalization by the number of steps s. As written, it sums per-step indicators over all videos and divides only by |D|, so the score would be an average number of correctly placed steps per video (up to s) rather than a percentage; the reported values (e.g., 24.1% for random, 98.3% for human) imply that the implementation also divides by s. Please correct the equation and state explicitly how step accuracy is averaged over steps and videos.","section":"§3.2, step accuracy definition"},{"comment":"All results are single-run point estimates with no error bars, confidence intervals, or significance tests. This matters because the paper uses 'significant' for RRD gains (e.g., §5.2) and because the evaluated closed-source models are stochastic; a gap such as 49.9 vs 46.6 in Table 3 may be within run-to-run noise. Please report mean and standard deviation over multiple generations or seeds for the key comparisons, and apply a paired significance test (e.g., McNemar or permutation) to the RRD gains. This is particularly important for the claim that adding video at the reasoning stage degrades performance.","section":"§4.2, Tables 1-5"}],"minor_comments":[{"comment":"The sentence 'The blank frames preceding the clips labeled chronologically' is unclear; please specify how clip numbers are displayed to the model (e.g., text overlay at the start of each clip) and confirm that the model can reliably perceive them.","section":"§3.1, Step 3"},{"comment":"The text reports gains of 15.3% for the 7B model and 20.8% for the 72B model, while Table 2 reports 15.4 and 20.9; please align the text and table.","section":"§5.2, text after Table 2"},{"comment":"The top-50 action and object panels are too small to read in the preprint; please provide tabular versions in the appendix or increase the font size.","section":"Figure 5(D,E)"},{"comment":"Please state explicitly how the random-guess baselines are computed, namely 1/k! for overall accuracy and the average per-position match for step accuracy, evaluated over the actual distribution of k in the benchmark rather than a single average value.","section":"§3.2, random baseline"},{"comment":"The statement that GPT-4o performs worst among closed-source models 'likely due to its limited capacity for long visual inputs' is presented as fact; please rephrase it as a hypothesis or support it with measurements of input-length handling.","section":"§4.2, GPT-4o discussion"},{"comment":"Human performance is based on eight volunteers and about 40% of the videos; please state whether each video was annotated by multiple evaluators and report inter-annotator agreement, since human accuracy is used to argue that the tasks are unambiguous.","section":"Appendix B.2, human evaluation"}],"recommendation":"major_revision","confidential_remarks":"This is a potentially useful benchmark contribution, but the headline interpretation overreaches relative to the evidence. I would ask the authors to run the goal-name and text-only-oracle control experiments before acceptance; if those controls show that ordering is largely recoverable from language, a substantial reframing of the central claim is needed. The paper would also benefit from a reviewer with specific expertise in video benchmark validation. No ethical concerns were identified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: VCRBench is a useful new probe, but the paper's headline claim—that LVLMs lack video-based causal reasoning—is overdetermined. The stress-test note is right: the question names the goal, and the ordering can be done from language-level commonsense once each clip is described, so the benchmark mostly tests open-ended visual event verbalization plus procedural knowledge, not causal reasoning performed directly over visual observations.\n\nWhat's genuinely new: the shuffled-clip ordering task over procedural videos is a fresh formulation. The construction pipeline is careful—CrossTask human timestamps, WikiHow step lists, manual curation, blank-frame boundaries, deterministic scoring. The 96.4% human accuracy shows the task is clear. The evaluation across 20+ models with a consistent pattern of near-random performance is credible, and several models just output consecutive numbers, a useful failure mode. RRD is simple and the gains are real: Qwen2.5-VL 72B goes from 29.0% to 49.9% with the basic two-stage prompt, and to 54.2% with sequential recognition. The ablation showing that adding video at the reasoning step hurts is an interesting finding, though it also cuts against the visual-causal interpretation.\n\nSoft spots, in order of importance. First, goal-name leakage: the benchmark does not isolate visual causal reasoning because the correct order of the described steps is recoverable from the named procedure plus commonsense. A control with no video, or with the goal name omitted, would clarify how much of the signal is visual. Second, no error bars or significance tests: several models sit near 0–3%, so noise matters, and the 25.2% improvement highlighted in the abstract comes from a different RRD configuration than the 20.9% in Table 2—the paper should state that explicitly. Third, the data and code are not actually linked despite the placeholder in the header; that is easy to fix but blocks reproducibility.\n\nBottom line: the benchmark is worth having and the main negative result is likely real. The paper should be revised to frame VCRBench as a multimodal-integration diagnostic rather than a pure test of visual causal reasoning. A serious referee should engage with it.","headline":"Useful new benchmark with a credible negative result, but the visual-causal interpretation is overstated because the goal name leaks and ordering can be done from language commonsense.","tokens_in":20507,"tokens_out":3666,"would_cite":true,"duration_ms":36262,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current large video language models cannot reconstruct the causal order of shuffled procedural steps, and most score at or below random guessing on a new benchmark that humans solve at 96.4%.","keywords":["causal reasoning","large video language models","VCRBench","benchmark design","procedural video understanding","sequence ordering","recognition-reasoning decomposition","language priors"],"falsifier":"Present the same shuffled-clip task using only the ground-truth text descriptions of each clip and no video, and compare accuracy with the video-condition scores; if text-only ordering matches or beats the roughly 29-58 percent video scores, the benchmark's signal is carried by language rather than visual causal inference. A second check is to remove the procedure name from the question template and see whether model accuracy drops toward the random-guess baseline.","tokens_in":19487,"feed_emoji":"🎬","tokens_out":11161,"duration_ms":90290,"temperature":0.7,"pith_summary":"This paper claims that today's large video language models (LVLMs) cannot perform long-form causal reasoning in video: shown shuffled clips of an everyday procedure and asked to reconstruct the correct order, most open-source models score at or below random guessing, and the best model evaluated reaches 58.0% while humans reach 96.4%. To support this, the paper introduces VCRBench, 365 test videos across twelve everyday procedures, designed so that the correct order cannot be gleaned from option wording and is graded deterministically. It also proposes Recognition-Reasoning Decomposition (RRD), which first asks a model to describe each clip in one sentence and then orders those descriptions using common sense; on the strongest open model, Qwen2.5-VL-Instruct72B, this raises accuracy from 29.0% to 54.2%, surpassing Gemini-1.5-Pro. A sympathetic reader would care because the result separates action recognition from causal reasoning and identifies connecting visual events into a goal-directed chain as the capability that is missing.","feed_headline":"Many video models score at or below chance on shuffled-step ordering","feed_subtitle":"The best of them reaches 58% where humans score 96.4% on a new causal-reasoning benchmark.","key_machinery":"VCRBench is the measuring instrument: 365 test videos across twelve everyday procedures, built by cutting CrossTask instructional videos at human-annotated timestamps, discarding steps that lack causal dependencies, shuffling the surviving clips, and requiring the exact chronological order as the answer. Random-guess accuracy sits at 7.8% because a full permutation must match, while step accuracy gives partial credit per clip. RRD is the intervention: its video-recognition stage prompts the model for a one-sentence description of each clip with no ordering, and its causal-reasoning stage feeds those descriptions plus the named goal to the model's language knowledge to produce the order. Ablations show that recognizing clips one at a time helps, while replacing the all-at-once ordering with pairwise merge-sort comparisons hurts because long-range causal structure is lost.","core_discovery":"The central claim is that video-based long-form causal reasoning—deciding which visual events must come before which others to reach a stated goal—is a distinct skill that current LVLMs have not acquired, and that it can be measured and partly recovered by task decomposition. VCRBench presents a model with three to seven shuffled clips, each showing one key step of a familiar procedure, and demands the exact permutation as the answer; this format keeps deterministic grading while removing the linguistic cues of multiple-choice QA. On this benchmark, most open-source LVLMs perform at or below the 7.8% random-guess baseline, several simply emit consecutive numbers as their answer, and the best model, Gemini-2.0-Flash-Thinking, reaches only 58.0% against 96.4% for humans. The paper's own intervention, RRD, splits the task into a recognition stage that produces a one-sentence description per clip and a reasoning stage that orders those text descriptions; applied to Qwen2.5-VL-Instruct72B it raises accuracy from 29.0% to 54.2%, a gain of up to 25.2%. The paper also reports that feeding the video back into the reasoning stage lowers accuracy, which it interprets as evidence that current models lean on language knowledge rather than visual causal analysis.","pith_inferences":["A direct extension would be to hide or replace the procedure name in the prompt; if accuracy collapses, the benchmark is being solved from the named goal plus language priors, and the visual signal is not load-bearing.","Porting the shuffled-step format to domains whose causal order is not common knowledge—laboratory procedures, mechanical repairs, unfamiliar crafts—would yield a harder diagnostic that language priors cannot satisfy.","The success of the text-ordering stage suggests a concrete training target: teach LVLMs to extract order-relevant causal structure from pixels directly, using the recognition stage as a scaffold rather than a permanent crutch.","Since humans approach perfect accuracy, the videos themselves carry enough information to solve the task; a model forced to order events without a goal name would test whether the recorded pixels are sufficient."],"forward_implications":["If VCRBench measures what it claims, strong LVLM scores on multiple-choice video benchmarks overstate visual understanding, since those formats can be solved from option wording alone.","Scaling and reasoning-specialized training help only partially: even Gemini-2.0-Flash-Thinking leaves nearly forty points of accuracy on the table relative to humans.","Decomposing a complex multimodal task into recognition followed by text-based reasoning is a cheap way to gain up to 25.2 accuracy points, letting an open model surpass Gemini-1.5-Pro.","The accuracy drop when video is re-added at the reasoning stage implies that visual input can actively conflict with these models' language priors and hurt rather than help.","Because accuracy falls steeply as the number of causal steps grows from three to seven, the benchmark identifies the length of the causal chain as the main difficulty driver."],"supporting_citations":[{"why":"Supplies the instructional videos and human-annotated key-event timestamps from which VCRBench clips are cut and shuffled.","marker":"[63]"},{"why":"The open-source model family that gives the best open results and serves as the main testbed for RRD.","marker":"[28]"},{"why":"Provides the closed-source baselines (Gemini-1.5-Pro and Gemini-2.0-Flash-Thinking) that set the top scores RRD is compared against.","marker":"[24]"},{"why":"The second open-source model family used to show RRD's gains generalize across architectures and sizes.","marker":"[26]"},{"why":"The prompting paradigm of splitting a complex task into sub-tasks that motivates RRD's two-stage design.","marker":"[62]"},{"why":"Supplies the step-accuracy metric that gives partial credit alongside exact-order accuracy.","marker":"[65]"},{"why":"The closed-source baseline whose near-random video performance supports the claim that limited long-video input hurts causal reasoning.","marker":"[79]"},{"why":"Representative multiple-choice video benchmark whose format VCRBench is explicitly designed to avoid to block linguistic shortcuts.","marker":"[16]"}],"fun_headline_variants":["Video models fail shuffled-step causal ordering","New benchmark exposes video reasoning gap: models near chance","Best video AI gets 58% on causal order, humans 96%","Decomposition boosts causal video reasoning by 25 points","Video models lean on language, not visuals, for causal order"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's claim to measure visual causal reasoning rests on the assumption that the correct clip order cannot be worked out from the named goal plus commonsense language knowledge alone, without genuinely analyzing the video.","fun_headline_variants_meta":{"raw":{"variants":["Video models fail shuffled-step causal ordering","New benchmark exposes video reasoning gap: models near chance","Best video AI gets 58% on causal order, humans 96%","Decomposition boosts causal video reasoning by 25 points","Video models lean on language, not visuals, for causal order"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000645,"raw_usage":{"total_tokens":3044,"prompt_tokens":1103,"completion_tokens":1941,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":719,"completion_tokens_details":{"reasoning_tokens":1861}},"tokens_in":719,"tokens_out":1941,"duration_ms":13020,"temperature":1.0,"reasoning_tokens":1861,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:54:04.989864+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Present the same shuffled-clip task using only the ground-truth text descriptions of each clip and no video, and compare accuracy with the video-condition scores; if text-only ordering matches or beats the roughly 29-58 percent video scores, the benchmark's signal is carried by language rather than visual causal inference. A second check is to remove the procedure name from the question template and see whether model accuracy drops toward the random-guess baseline.","supporting_citations":[{"cited_title":"Procedure planning in instructional videos","cited_arxiv_id":null,"evidence_quote":"Supplies the step-accuracy metric that gives partial credit alongside exact-order accuracy."},{"cited_title":"Mvbench: A comprehensive multi-modal video understanding benchmark","cited_arxiv_id":null,"evidence_quote":"Representative multiple-choice video benchmark whose format VCRBench is explicitly designed to avoid to block linguistic shortcuts."}],"review_version":1}