{"id":"0056d92a-7929-4bd7-b40a-f1e711ceecbd","arxiv_id":"2411.13211","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ViSTa is a new hierarchical video benchmark showing that vision-language models recognize objects well but fail to understand action order in sequential tasks.","lead":"This paper introduces ViSTa, a dataset of over 4,000 videos with step-by-step task descriptions in virtual home, Minecraft, and real-world settings. It tests whether vision-language models can judge sequential tasks and finds that current models, including GPT-4o, struggle with action order.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Virtual-home videos are stitched from different ALFRED episodes; the paper's main evidence that VLMs fail sequential tasks may reflect stitch artifacts rather than genuine VLM limitations, and no control verifies this.","rationale":"The reader's conditional verdict is well founded, and I agree with the identified weakest assumption. The paper's strongest claim—that current VLMs fail to understand sequential tasks—rests mainly on the virtual-home results, especially the monotonic performance drop with level in Figs. 3b and 4. Those videos are automatically assembled from clips of different ALFRED episodes, and Section A.1.2 explicitly admits visual discontinuities. The paper's justification that low frame rates make the glitches unproblematic is an assumption, not a measurement; low frame rates can make cross-episode stitches more salient when sampled frames show different rooms or different held objects, and they cannot repair violations of object identity across steps. Without a human baseline, a continuous re-rendering control, or a stitch-consistency audit, the validity of the dominant environment's sequential-task labels is unknown. The paper does deserve credit for its hierarchical dataset design, the open repository, detailed prompt documentation, and the manually validated real-world videos, but those do not rescue the virtual-home results. The proposed re-rendering test would directly settle whether the reported failure is a property of the models or of the data construction, so the conditional verdict should remain unchanged.","tokens_in":14804,"tokens_out":5330,"duration_ms":66260,"concrete_test":"Re-render a sample of 20–30 virtual-home permutation/remix problem sets as continuous ALFRED/THOR episodes executing the identical action sequences, then rerun the GPT-4o pipeline (and a small human baseline) on both the original stitched videos and the re-rendered videos. If accuracy on re-rendered videos is materially higher than on stitched originals, the Fig. 4 level drop is a stitching artifact; if accuracy is comparable, the artifact concern is discharged.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ViSTa's dominant virtual-home portion (levels 2–8) is generated by stitching short clips from different ALFRED episodes (Sec. 3.2, A.1.2). Section A.1.2 concedes sudden teleportation, changes in the held object, and scene inconsistencies, then asserts these are 'not problematic for our evaluations, since the models all have low frame-rates.' That assertion is unverified and carries the central argument. Low frame rate does not fix a semantic mismatch: if the 'pick up banana' clip and the 'put banana in closet' clip come from different episodes, the video may depict two unrelated bananas, so the ground-truth description is not actually entailed by the video as a continuous trajectory. Conversely, when stitched frames jump between different source scenes, the transition can make segment boundaries salient, leaking order information without requiring sequential understanding. Either way, the level-dependent drop in Fig. 3b/4, the main support for 'fail to understand sequential tasks,' could be an artifact of video construction. No human baseline, no continuous re-rendering control, and no stitch-consistency check is provided for the virtual-home data; the manual 100%-human-accuracy validation is stated only for real-world videos. The concern is not that the authors are careless, but that a specific unmeasured property of the majority of the dataset is load-bearing for the headline conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ViSTa, a hierarchical dataset of over 4,000 videos with step-by-step task descriptions in virtual home, Minecraft, and real-world environments. The dataset is organized into problem sets and levels, enabling evaluation of vision-language models on single-action and multi-action sequential tasks. The authors evaluate CLIP, ViCLIP, and GPT-4o on video-description matching, finding that all models perform well on object recognition but poorly on action-order understanding, with GPT-4o showing non-trivial but declining performance on longer sequences. The central claim is that current VLMs are not yet capable of supervising sequential tasks, with implications for using VLMs as process reward models in reinforcement learning.","tokens_in":15111,"tokens_out":2795,"duration_ms":29600,"significance":"If the results hold, ViSTa would be a valuable benchmark for studying VLM capabilities on process-based supervision, complementing existing outcome-based reward-model evaluations. The hierarchical structure and the separation of object recognition, object properties, actions, general sequential understanding, and permutation-based order understanding are thoughtful design choices. The paper also tests models across three environments, including real-world videos that are human-validated, and provides open-source code and data. The strongest evidence is the consistent near-baseline performance of CLIP and ViCLIP on permutation problems and the decline of GPT-4o with level, which would support the conclusion that current VLMs cannot reliably supervise multi-step tasks.","major_comments":[{"comment":"The virtual-home multiple-action videos are created by stitching clips from different ALFRED episodes, and the paper asserts that the resulting 'small visual glitches' are 'not problematic for our evaluations, since the models all have low frame-rates.' This assertion is load-bearing because the majority of ViSTa consists of these stitched videos. A low frame rate does not remove semantic mismatches (e.g., a pick-up clip and a put-down clip of a banana from different episodes can depict different objects), nor does it prevent stitch boundaries from making segment order artificially salient. No control is provided: there is no comparison between stitched videos and continuous re-renderings of the same tasks, no human baseline on the virtual-home data, and no analysis of whether model errors concentrate on videos with detectable discontinuities. Without such a control, the level-dependent drop in Fig. 3b and Fig. 13a may reflect video-construction artifacts rather than model limitations in understanding action order.","section":"3.2 and A.1.2"},{"comment":"Human validation at 100% accuracy is claimed only for the real-world videos, not for the virtual-home or Minecraft data. Since the virtual-home videos are automatically generated and constitute the majority of the dataset, the ground truth for those videos is not independently verified. If the stitched videos are ambiguous even to humans, the measured 'failure' of VLMs would be an artifact of the dataset rather than a model deficiency. The paper should provide a human baseline on a sample of virtual-home videos (and ideally Minecraft) to establish that the intended task is actually entailed by the videos.","section":"A.1.4 and Author contribution statement"}],"minor_comments":[{"comment":"The phrase 'they fail to understand sequential tasks' is stronger than the reported results, which show GPT-4o achieving non-trivial (though declining) performance on permutation problems, while CLIP and ViCLIP are near baseline. Consider softening the wording to 'models are not yet reliable at supervising sequential tasks' to match the evidence.","section":"Abstract and Section 5"},{"comment":"The score standardization computes µ and σ from the evaluation videos themselves, which could in principle leak information about the test set. The footnote says results did not meaningfully differ when excluding the video being tested, but this leave-one-out variant should be reported in the main text or appendix to make the protocol unambiguous.","section":"Section 4.1, Eq. (1)"},{"comment":"The claim that 'real videos are easier than simulations' rests on a comparison of only 9 real-world videos against an unspecified number of virtual-home videos. This is a small sample, and the conclusion should be hedged accordingly.","section":"Section 4.2 and Figure 17"},{"comment":"The attribution of CLIP-8's advantage over ViCLIP to 'larger scale' is speculative, since the models differ in architecture, training data, and other factors beyond parameter count. Please phrase this as a hypothesis rather than a conclusion.","section":"Section 4.2 and Figure 5"},{"comment":"The prompt text in Figure 10 contains a typo ('sligthly' should be 'slightly').","section":"Appendix C, Figure 10"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the virtual-home stitching artifacts could invalidate the headline claim. The authors should be asked to provide a human baseline on virtual-home data or a continuous-video control before the paper is published; otherwise the benchmark itself remains a contribution, but the stated conclusions about VLM sequential understanding are not yet supported. The paper is otherwise well within the scope of a computer vision / multimodal evaluation venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dataset is the real contribution here. The hierarchical structure, where single actions compose into longer sequences, and the split into general vs. permutation problem sets, is a clean way to separate object recognition from action-order understanding. The real-world videos, especially the mimic set, are a thoughtful addition for comparing simulation to reality. On the empirical side, the result that CLIP and ViCLIP sit at or near baseline on permutation tasks, and that GPT-4o degrades as level increases, is credible and matches the prior work the paper cites (Paxion). Credit where due: this is a solid benchmark artifact that the community can build on.\n\nThe soft spot is real and load-bearing. Section A.1.2 concedes that the virtual-home videos, which are the majority of the dataset, are generated by stitching clips from different ALFRED episodes, with teleportation, changes in held objects, and scene inconsistencies. The paper asserts these glitches are fine because the models have low frame rates, but that is not an argument. Low frame rate does not fix a semantic mismatch: if the 'pick up banana' clip and the 'put banana in closet' clip come from different episodes, the video may simply be showing two unrelated bananas, so the ground-truth description is not entailed by the video. Conversely, the stitch boundaries themselves can make the segment order visible, leaking the answer without requiring sequential understanding. Either way, the level-dependent drop in the main figures—the central evidence for 'fail to understand sequential tasks'—could be an artifact of how the videos were constructed. The paper provides no human baseline for virtual home or Minecraft, and no control that uses continuous, unstitched videos. The claim that glitches are 'likely undetectable' is an assumption that the evaluation itself depends on.\n\nOther concerns are minor. The abstract overstates the failure: GPT-4o achieves non-trivial performance, and the real-world results show no large drop with level, so 'fail to understand' is too broad. Standardizing scores using statistics computed on the evaluation videos is a slight protocol leak, but the authors report it doesn't change results.\n\nWho should read this: anyone working on VLM reward models for RL or on video understanding benchmarks. It deserves a serious referee, but the revision needs to address the stitching artifact, add human baselines for the simulated environments, and soften the conclusion. Without those, the dataset is still usable, but the headline finding is not established.","headline":"ViSTa is a genuinely useful benchmark for testing sequential understanding in VLMs, but the headline claim is not fully supported because the virtual-home videos are stitched from different ALFRED episodes and no control verifies that stitching artifacts aren't driving the order-related failures.","tokens_in":15601,"tokens_out":1465,"would_cite":true,"duration_ms":17287,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that current vision-language models, including GPT-4o, cannot reliably understand multi-step sequential tasks, despite strong object recognition.","keywords":["vision-language models","reward models","sequential task understanding","action order","video-language benchmark","reinforcement learning","process supervision"],"falsifier":"Have human annotators watch the automatically stitched virtual home permutation videos and try to identify the correct action order. If humans cannot reliably do so, the videos do not contain the order information ViSTa claims to measure, and the reported model failures would reflect dataset artifacts rather than a lack of sequential understanding.","tokens_in":14643,"feed_emoji":"🎬","tokens_out":6739,"duration_ms":67261,"temperature":0.7,"pith_summary":"The paper asks whether vision-language models (VLMs) can serve as reward models for tasks that cannot be scored by the final state alone, meaning whether they genuinely understand the step-by-step order of a task. It introduces ViSTa, a dataset of more than 4,000 videos in virtual home, Minecraft, and real-world environments, organized into eight levels of increasing sequential complexity. Evaluations of CLIP, ViCLIP, and GPT-4o show that all three recognize objects well but struggle with sequences: on permutation problems, where only the order of actions differs, GPT-4o's performance drops sharply as videos lengthen, while the other models stay near the majority-class baseline. The paper concludes that current VLMs are not yet capable of supervising any but the most basic tasks.","feed_headline":"VLMs can't judge multi-step tasks, 4,000-video test shows","feed_subtitle":"Only GPT-4o beats the baseline on action order, and its accuracy halves on eight-step videos.","key_machinery":"ViSTa's hierarchical structure is the central mechanism: level-1 single-action videos are composed into levels 2 through 8, and video-description pairs are grouped into problem sets that test object recognition, object properties, action understanding, and, through permutation problem sets, action order. The scoring protocol samples 8 to 32 frames per video, standardizes softmax scores across classes, and picks the highest-scoring description. The permutation problem sets are the load-bearing part of the benchmark because they keep objects and actions the same and vary only their order.","core_discovery":"On ViSTa's permutation problem sets, the only tested model that beats the majority-class baseline is GPT-4o, and its macro F1 begins to fall after level 4, ending at roughly half of its original value for eight-action videos. CLIP and ViCLIP stay barely above baseline throughout. In general multi-step problem sets, CLIP can match videos to descriptions without any sense of order, which the paper attributes to the models relying on object recognition rather than on action order. The paper's central claim is that current VLMs cannot yet supervise sequential tasks and lack robust action-order understanding.","pith_inferences":["A testable extension is to fine-tune a VLM on order-contrastive pairs like ViSTa's permutation sets; success on held-out environments would suggest the failure is a training gap rather than a fundamental architectural limit.","The finding that general multi-step problems can be solved without order understanding implies that earlier VLM reward-model successes on trajectory-style tasks may have been driven by object recognition; this could be checked by testing those reward models on wrong-order videos with identical objects.","Because the glitch concern applies to the largest split, the paper's strongest evidence for the conclusion comes from the smaller real-world and Minecraft sets; scaling those up would directly test whether the virtual-home results generalize."],"forward_implications":["VLM-based process reward models are not yet viable for reinforcement learning on multi-step tasks, because the model cannot reliably tell whether steps happened in the correct order.","Evaluation of task understanding must separate order understanding from object recognition; ViSTa's permutation problem sets do this, while general problem sets do not.","Improving frame rate or model scale may improve general task matching, but the permutation results show that these levers do not fix order understanding for the tested models.","For now, a frontier model like GPT-4o can be considered for supervision only of single-step or very short tasks, and its drop on eight-action videos shows why.","If the failures are general, benchmark designers should treat 'matches the objects' and 'understands the sequence' as distinct capabilities rather than one skill."],"supporting_citations":[{"why":"Supplies the virtual home clips that are re-mixed into most ViSTa videos.","marker":"[24]"},{"why":"Established zero-shot CLIP reward models, the approach this paper extends to sequential tasks.","marker":"[8]"},{"why":"Motivated process-based supervision and revealed failures in video-language behavior critics.","marker":"[10]"},{"why":"Showed that VLMs rely heavily on object recognition, which the paper uses to explain general-task success.","marker":"[19]"},{"why":"Provides the ViCLIP model that is one of the three evaluated VLMs.","marker":"[31]"},{"why":"Documents GPT-4o, the frontier model that achieves the paper's only non-trivial results.","marker":"[32]"},{"why":"Provides the CLIP model evaluated as a frame-averaged video scorer.","marker":"[28]"},{"why":"Supplies the 200 door open/close videos used to test action recognition in complex contexts.","marker":"[25]"},{"why":"Supplies the Minecraft skill set used to define single-action videos.","marker":"[27]"}],"fun_headline_variants":["VLMs fail at judging task order in 4,000-video test","Only GPT-4o beats baseline on action order, then drops","Vision-language models miss sequential steps, rely on objects","Sequential tasks trip up VLMs, only GPT-4o partially works"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the automatically stitched virtual home videos in ViSTa, which contain visual glitches such as sudden teleportation and changes in held objects, still carry valid, human-recoverable action-order information; the paper does not verify this.","fun_headline_variants_meta":{"raw":{"variants":["VLMs fail at judging task order in 4,000-video test","Only GPT-4o beats baseline on action order, then drops","Vision-language models miss sequential steps, rely on objects","Sequential tasks trip up VLMs, only GPT-4o partially works"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000246,"raw_usage":{"total_tokens":1492,"prompt_tokens":850,"completion_tokens":642,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":575}},"tokens_in":466,"tokens_out":642,"duration_ms":6351,"temperature":1.0,"reasoning_tokens":575,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:41:53.895494+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators watch the automatically stitched virtual home permutation videos and try to identify the correct action order. If humans cannot reliably do so, the videos do not contain the order information ViSTa claims to measure, and the reported model failures would reflect dataset artifacts rather than a lack of sequential understanding.","supporting_citations":[{"cited_title":"Paxion: Patching Action Knowledge in Video-Language Foundation Models,","cited_arxiv_id":null,"evidence_quote":"Showed that VLMs rely heavily on object recognition, which the paper uses to explain general-task success."}],"review_version":1}