{"id":"6935364a-56d6-49e6-949a-64408feb505a","arxiv_id":"2607.13527","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"VGIF-Score decomposes video prompts into dependency graphs and uses a VLM to diagnose which instruction constraints models satisfy, revealing strong failures on causal and late-prompt constraints.","lead":"This paper introduces VGIF-Score, a two-part automated system for grading how well video-generation models follow long, complex instructions: one part checks concrete objects and events, the other rates visual quality. It also releases VGIF-Bench, a benchmark of 223 challenging prompts, and reports that current models fail badly on causal and late-prompt requirements.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Depth-sensitivity finding is partly an artifact of the recursive short-circuit scoring in Eq. (2): Fig. 3b's decline is manufactured by the propagation rule, not necessarily by model behavior.","rationale":"The reader's weakest assumption focused on the single-VLM evaluator's reliability; that is a valid external reproducibility concern. The concern I raise is internal to the scoring methodology and is more directly tied to the central claim about discovering systematic failure modes. Even if the VLM is perfectly calibrated, Eq. (2) mixes inherited failures with per-node failures, so the reported depth and causal trends cannot be cleanly attributed to model limitations. This does not invalidate the framework entirely—the short-circuit design is useful for localizing failures, as in Figure 4—but it means the aggregate structural analysis overinterprets the data. The paper should present both propagated and unpropagated accuracy to distinguish 'downstream success' from 'per-node capability.' This is a concrete, addressable issue, so a conditional verdict is appropriate: the current evidence is suggestive but not decisive for the depth-sensitivity claim.","tokens_in":11976,"tokens_out":4734,"duration_ms":54053,"concrete_test":"Recompute Fig. 3b and the Causal column of Table 2 using an unpropagated per-node correctness c'_i = 1[answer_i = a_i] (drop the dependency conjunction in Eq. (2)) while keeping the same QA answers. Also report the conditional accuracy for depth-d nodes given that all ancestors are correct. If the monotone decline from depth 0→4 and the causal deficit largely persist in c', the depth-sensitivity finding is robust to the scoring artifact. If the decline flattens or the causal gap shrinks substantially, the reported failure modes are dominated by short-circuit propagation and must be reinterpreted as end-to-end success rather than per-node capability.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's core diagnostic claim—'strong sensitivity to dependency depth'—is supported by Fig. 3b, which plots accuracy as a function of ST-DAG depth. But Eq. (2) defines per-node correctness as ci = 1[answer_i = a_i] AND eval(dep(i), {c_j}_{j<i}). Consequently, a node at depth d is marked incorrect whenever any ancestor failed, even if the VLM correctly answered the node's own question. The plotted 'accuracy at depth d' is therefore not an estimate of the model's per-node capability at depth d; it is the joint probability that the node and all its ancestors are correct. Under any evaluator with per-question error > 0, monotone decline with depth is guaranteed by the scoring rule regardless of model behavior. Indeed, the depth 0→1 drop (80.6→58.5) is close to 0.806^2 ≈ 0.65, consistent with mechanical propagation. The same confounding inflates the reported causal bottleneck, because causal nodes are typically downstream of prerequisite actions/states and are short-circuited when prerequisites fail. Thus the headline failure modes may be at least partially artifacts of the aggregation rule, and the paper's claim that VGIF-Score 'reveals' these failure modes is not yet supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VGIF-Score, a framework for evaluating instruction following in text-to-video generation. Prompts are parsed by an LLM into a Spatio-Temporal Directed Acyclic Graph (ST-DAG), from which dependency-aware binary QA pairs are derived and answered by a VLM. A second subjective branch uses an instruction-conditioned AutoRubric to score cinematography, visual purity, motion smoothness, and physics adherence. The two branches are equally weighted into a final score. The authors instantiate the framework on VGIF-Bench (223 prompts, ~4.3K evaluation items), evaluate 14 proprietary and open-source video generation models, and report two systematic failure modes: weak causal instruction following and strong sensitivity to dependency depth and prompt position. Human validation on 200 videos indicates high agreement with the VLM evaluator.","tokens_in":12254,"tokens_out":3520,"duration_ms":41635,"significance":"If the claims were fully supported, VGIF-Score would be a valuable contribution: it moves video-generation evaluation from holistic, aggregate scoring toward fine-grained, diagnostic, and interpretable assessment, and it introduces a dependency-aware graph structure that is not present in prior benchmarks such as VBench, T2V-CompBench, or ChronoMagic. The explicit ST-DAG representation and the attempt to localize failures to specific dependency edges are genuinely useful design ideas. The benchmark construction pipeline, which combines LLM drafting with human verification, is also practical and scalable. However, the central diagnostic findings — depth sensitivity and, to a significant degree, the causal bottleneck — are confounded by the short-circuit propagation rule in Eq. (2), and the entire evaluation rests on a single proprietary VLM with only a limited human-validation sample. The interpretive value of the paper is therefore not yet established.","major_comments":[{"comment":"The headline finding of 'strong sensitivity to dependency depth' is largely an artifact of the scoring rule. Eq. (2) defines c_i = 1[answer_i = a_i] AND eval(dep(i), {c_j}_{j<i}), so a node at depth d is marked incorrect whenever any ancestor is incorrect, even if the VLM correctly answers that node's own question. The 'accuracy at depth d' plotted in Fig. 3b is therefore the joint probability that the node and all its ancestors are correct, not a per-node capability estimate. Under any evaluator with per-question error rate epsilon > 0, accuracy is guaranteed to decline monotonically with depth. The observed depth 0→1 drop from 80.6% to 58.5% is close to 0.806^2 ≈ 0.65, consistent with pure propagation. The same mechanism inflates the causal bottleneck in Table 2, since causal nodes are typically downstream of prerequisite actions/states. To support the claimed diagnosis, the authors mu","section":"Section 3.2, Eq. (2) and Section 5.4, Fig. 3b"},{"comment":"The prompt-position effect is confounded with node type and dependency depth. The paper states that accuracy drops from 67.9% in the first 20% of the prompt to 10.1% in the final 20%, and later that 'position sensitivity is most severe for semantically complex constraints' — but no control is reported for the fact that later positions likely contain more state, causal, and deeply dependent nodes. Without stratifying by node type and depth (or performing a regression with those covariates), the conclusion that position and depth are 'two orthogonal axes' is not established. The authors should report, e.g., position curves within each node type and within fixed depth bands.","section":"Section 5.4, Fig. 3a"},{"comment":"All benchmark conclusions, including all model rankings, error analyses, and structural findings, are produced by a single proprietary VLM (Gemini-3.1-Pro). The human validation uses only 200 videos, with no breakdown by model, category, depth, or question type, and no analysis of the VLM's error patterns (e.g., false positives vs. false negatives on QA, or ceiling/floor effects). The reported aggregate agreement (96.3%, κ=0.92) is encouraging, but it does not demonstrate that the VLM's judgments are reliable enough to support the specific depth/position/causal claims, which involve small differences at extreme ranges. The authors should provide per-model or per-stratum human agreement, a second evaluator cross-check, or at least an error analysis of the VLM on the validated sample.","section":"Section 5.6 and Section 5.1"}],"minor_comments":[{"comment":"Typo: 'depencey-depth' should be 'dependency-depth'.","section":"Abstract and Introduction"},{"comment":"VGIF-Bench contains only 223 prompts, far fewer than several comparison benchmarks. The paper justifies this by structural complexity, but a discussion of potential selection bias and statistical power would be useful.","section":"Table 1"},{"comment":"The table reports 'VGIF-Score 0.71 0.83 0.89' without explicitly stating which columns are Spearman correlations and which are Pearson. It later says 'Spearman rank correlation' for rating-based scores, but the table caption should be self-contained.","section":"Table 4"},{"comment":"The text says the position effect is 'universal across all 14 VGMs', but only aggregate numbers and one example (PixVerse-V6) are shown. A per-model plot or supplementary table would support this strong claim.","section":"Section 5.4, Fig. 3a"},{"comment":"The notation eval(dep(i), {c_j}_{j<i}) is explained in prose but would benefit from a formal definition of the Boolean evaluation function, especially for OR dependencies with multiple parents.","section":"Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The core evaluation framework is well specified and the benchmark construction is careful, but the key diagnostic claims are currently confounded by the short-circuit propagation rule. The paper is likely salvageable: if the authors re-analyze the data with conditional or raw per-node accuracy and address the position confound, the findings would be substantially stronger. The reliance on a single proprietary VLM is also a concern for a benchmark meant to be widely used; the authors should provide more extensive validation evidence or make the evaluator swappable. I would not reject the paper, but the current version does not support its headline conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: this is a genuine contribution to diagnostic evaluation of instruction following in video generation, but the paper's headline finding about dependency depth is inflated by its own scoring rule, so the depth curves need to be read with care.\n\nVGIF-Score does two things well. The ST-DAG decomposition turns a prompt into explicit atomic constraints with dependency edges, which is a real step beyond treating prompts as flat text. The dependency-aware QA with short-circuit propagation is a sensible way to localize failures along causal chains, and the instruction-conditioned AutoRubric adds a perceptual check that most benchmarks lack. The benchmark construction is transparent and the human validation on 200 videos, while not huge, is respectable. The empirical sweep across 14 models is useful.\n\nThe soft spot is the depth analysis. Eq. (2) defines a node as correct only if its own answer is right and all its dependencies are satisfied. So the accuracy plotted in Fig. 3b at depth d is the joint probability that the node and every ancestor are correct. Under any evaluator with per-question error above zero, this decreases with depth even if a model follows each individual instruction with exactly the same probability. The 80.6→58.5 drop from depth 0 to 1 is roughly consistent with mechanical propagation (0.806² ≈ 0.65). If you condition on the parent being correct, the model's own success rate at depth 1 is about 72.6%, so there is a real but much smaller effect. The paper presents the 22.1-point drop as \"the immediate cost of even a single dependency,\" which overstates the case. The same propagation inflates the causal bottleneck, since causal nodes sit downstream. The qualitative conclusion that current VGMs struggle with causal instruction is probably still directionally right, but the paper doesn't provide the raw per-node accuracy without short-circuiting, so I can't fully separate the real failure from the scoring artifact.\n\nThe other concern is the single proprietary VLM. The entire ranking rests on Gemini-3.1-Pro, and human validation covers only 200 videos. The artifacts aren't released in this version. I'd want to see the data/code and some evidence on evaluator stability. The equal-weight aggregation is also untested, though that's minor.\n\nWho should read this: people building video-generation benchmarks or working on compositional generation. It deserves peer review, but the authors should redo the depth analysis (e.g., condition on ancestors, report raw answer accuracy), release the artifacts, and add evaluator robustness checks. The ST-DAG representation is worth citing.","headline":"ST-DAG-based evaluation is a genuine step forward, but the headline depth-sensitivity result is partly an artifact of the short-circuit scoring rule.","tokens_in":12771,"tokens_out":5082,"would_cite":true,"duration_ms":55485,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Video instruction following is best measured by decomposing prompts into dependency graphs and propagating failures along causal chains, which reveals that current models collapse on deep and late constraints.","keywords":["video generation evaluation","instruction following","spatio-temporal dependency graph","dependency-aware QA","short-circuit diagnostics","video generation benchmark","causal reasoning","VLM-based evaluation"],"falsifier":"Run the full 223-prompt, 4.3K-item protocol twice with two different strong vision-language evaluators, or with complete human annotation of all items, and compare per-dimension and per-model scores; if the 14-model ranking, the 6.7-fold position decline, or the depth falloff changes materially across evaluators, the reported failure modes are evaluator artifacts rather than generator properties.","tokens_in":11853,"feed_emoji":"🎬","tokens_out":6245,"duration_ms":64703,"temperature":0.7,"pith_summary":"The paper's central claim is that instruction following in video generation can be measured at the level of individual semantic constraints, and that doing so exposes failures that aggregate quality scores hide. To show this, the authors introduce VGIF-Score, which parses each prompt into a spatio-temporal directed acyclic graph (ST-DAG) of entities, actions, states, and causal relations; verifies each node through binary question-answering against the generated video; and propagates failures along dependency edges via a short-circuit rule. This objective branch is paired with a per-prompt auto-generated rubric for cinematography, visual purity, motion smoothness, and physics adherence, and both components are checked against human judgments on a 200-video sample. The framework is instantiated as VGIF-Bench, containing 223 long, dependency-rich prompts that yield about 4.3K fine-grained evaluation items, and applied to 14 video generation models. The paper's main empirical finding is that current models rarely follow causal constraints, and that accuracy collapses both with dependency depth (roughly 19 percentage points per level) and with constraint position in the prompt (a 6.7-fold drop from the first fifth to the last fifth).","feed_headline":"New metric shows video models drop 6.7x on late prompt cues","feed_subtitle":"A prompt-to-graph evaluator localizes failures to causal chains that aggregate scores hide.","key_machinery":"The load-bearing object is the Spatio-Temporal Directed Acyclic Graph (ST-DAG): each node is an atomic semantic unit—entity, attribute, location, action, state, or causal—and each directed edge is a dependency, either a compositional prerequisite or a causal consequence. The graph is turned into binary QA pairs, and per-node correctness is computed recursively: a node is credited only if the evaluator's answer is correct and the node's dependency expression (using AND/OR connectives over upstream nodes) is satisfied. This short-circuit rule is what converts a flat accuracy number into a failure-localization tool: once a prerequisite fails, every downstream node is automatically marked incorr","core_discovery":"On its own terms, VGIF-Score establishes that long, compositional video instructions can be decomposed into a dependency graph and evaluated automatically, constraint by constraint, with results that align with human judgments: 96.3% agreement on the QA items, a rank correlation of 0.87 between the combined score and human overall ratings, and stronger human alignment for the combined score than for either branch alone. Applied to 14 proprietary and open-source models on more than 3,000 generated videos, the framework finds that the bottleneck in video generation is not visual fidelity but structured execution: causal-relation accuracy stays below about 4.2 for commercial models and near zer","pith_inferences":["An extension the paper leaves implicit: because the short-circuit rule zeroes every downstream node once an upstream gate fails, the score measures minimal achievable completion, not partial credit; a future variant could report both to distinguish 'never attempted' from 'attempted but wrong'.","The same protocol could be turned into a training signal: ST-DAG failure traces identify which prompt segments a generator ignores, suggesting a way to reweight late and deep clauses in reinforcement learning or reward modeling.","A direct stress test of the paper's position finding would shuffle clause order while keeping the dependency graph fixed; the paper reports position sensitivity but does not separate lexical recency from structural ordering, so a shuffle experiment could clarify the cause of the 6.7-fold decline.","The dependency-aware evaluation assumes the vision-language evaluator can perceive cause-effect in video; a supplementary test using temporally edited clips—removing a causal event while keeping the rest—would check whether the evaluator's causal answers truly track event presence."],"forward_implications":["If the paper is right, evaluations of text-to-video models should report position- and depth-stratified accuracies, because aggregate scores mask a 6.7-fold recency cliff and a roughly 19-point-per-depth-level falloff.","Causal chains are the binding constraint: on the paper's numbers no tested model exceeds a causal QA score of 4.21, so progress in instruction following should be tracked on causal and state dimensions rather than entity or location accuracy.","A single failed prerequisite can invalidate all downstream constraints, so diagnostic dashboards built on short-circuit propagation can point to the exact gating edge that needs improvement in a model.","Combining objective QA with a per-prompt rubric correlates better with human overall judgment (0.87) than either branch alone, supporting the paper's dual-branch design as a standard evaluation protocol.","Benchmark prompts need deep dependency structures—VGIF-Bench averages 16.4 nodes and 17.7 edges per prompt—to separate models that merely render scenes from models that execute event sequences."],"fun_headline_variants":["Video models fail long prompts: new metric localizes the break","Causal chains trip up video generators, per VGIF-Score","VGIF-Score: pinpointing where video generation loses the plot","Not visual fidelity: structured execution is video models' weak spot"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"A single proprietary vision-language model is assumed to answer the binary ST-DAG questions and rubric ratings accurately enough that the reported model rankings, the 6.7-fold position cliff, and the depth falloff reflect the generators rather than the evaluator's own biases or instability; the supporting human check covers 200 sampled videos, not the full benchmark.","fun_headline_variants_meta":{"raw":{"variants":["Video models fail long prompts: new metric localizes the break","Causal chains trip up video generators, per VGIF-Score","VGIF-Score: pinpointing where video generation loses the plot","Not visual fidelity: structured execution is video models' weak spot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1268,"prompt_tokens":800,"completion_tokens":468,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":394}},"tokens_in":544,"tokens_out":468,"duration_ms":4688,"temperature":1.0,"reasoning_tokens":394,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T04:52:55.516743+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full 223-prompt, 4.3K-item protocol twice with two different strong vision-language evaluators, or with complete human annotation of all items, and compare per-dimension and per-model scores; if the 14-model ranking, the 6.7-fold position decline, or the depth falloff changes materially across evaluators, the reported failure modes are evaluator artifacts rather than generator properties.","supporting_citations":[],"review_version":1}