{"id":"0f1c00ba-8b77-42e3-b9e6-3443895a32fe","arxiv_id":"2607.29401","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new benchmark and model for cross-video procedure planning where evidence retrieval, window localization, and action planning are scored jointly; OSEF improves video-and-plan success over hard-selection baselines.","lead":"This paper introduces a new video task where a model must find the right demo video, find the right moment, and predict the steps, all from a start-goal text query. It also presents a method that keeps all candidate videos in play during planning and reports gains on video-plus-plan accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The fail-closed leakage audit in Sec. 3 is under-specified: it checks only exact phrase matches against action labels, yet the deterministic query templates may still encode the answer via paraphrase, so all retrieval and planning scores could be inflated.","rationale":"Assessed candidate objections: (a) FULL-SR excludes localization - important but orthogonal to the ranking claim because the paper explicitly separates Evidence and Plan axes and reports FULL@IoU scores; it is a design choice, not an internal inconsistency. (b) Validation-selected tables - robustness is supported by the locked-split retrain and large margins on native cells. (c) Self-built SOTA baseline - the component study partially addresses this. The leakage audit is the one link that, if broken, invalidates all experimental numbers regardless of other strengths. The paper's own Sec. 6.6 controls are only partial evidence; the audit covers all sources and all templates, but the description is too thin to rule out paraphrase leakage. An independent text-only probe is the decisive test. Since the reader already identified the audit as the weakest assumption, the verdict remains CONDITIONAL with no change.","tokens_in":15926,"tokens_out":11075,"duration_ms":126269,"concrete_test":"Take a random sample of 500 T3 examples from COIN and CrossTask. Run (1) a text-only retrieval baseline: e.g., BM25 or a sentence transformer that matches the legal query against the ASR/captions or state-change annotations of the 4 candidate videos, and measure R@1. (2) A text-only plan predictor: fine-tune a small language model to output the action sequence from the query alone (no video features) and compute FULL-SR. If text-only retrieval R@1 substantially exceeds the majority-sequence floor, or if text-only plan FULL-SR approaches OSEF's same-task COIN-T3 result of 18.91%, then the fail-closed audit is insufficient and the benchmark's leakage gate must be strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is the Sec. 3 fail-closed audit: 'a normalized phrase audit compares every query against its target action labels, and every match is human-adjudicated or quarantined.' This only checks exact or substring matches against action labels. But Sec. 3 also states queries 'follow deterministic templates' derived from start-goal state descriptions. If those state descriptions are extracted from the same video annotations that define the hidden action sequence (e.g., COIN state-change labels), the legal query may encode the answer in paraphrase—e.g., the goal 'The melted soap block is transferred into the vessel' is a near-paraphrase of the action 'transfer soap into vessel.' A model with a text-only head could exploit this to predict the plan or retrieve the correct video by n-gram overlap with the video's captions, without using cross-modal evidence. The paper's Sec. 6.6 controls (query-only FULL-SR 1.12%, wrong-transition R@1 drop only 1.81 pts) are encouraging but are limited to one hard same-task COIN cohort; they do not rule out leakage in other sources (e.g., NIV, CrossTask) or against stronger text-only baselines like fine-tuned language models. Because every retrieval and planning score in Tables 3-4 depends on the query not leaking the answer, this is the least secure link in the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Cross-Video Scene Procedure Planning (CVSPP), a task in which a model receives an answer-redacted start–goal text query and a set of K candidate videos, and must retrieve the supporting video, localize the relevant temporal window, and predict the intervening closed-set action sequence. The authors construct an 11-source benchmark with typed negative roles, a fail-closed query-observability audit, and separate Evidence- and Plan-axis metrics. They adapt nine existing VSPP planners and propose OSEF, which scores a query-conditioned cell-and-span lattice over all candidates and feeds the full soft lattice to a planner through a token-global adapter, avoiding an early hard video/window selection. OSEF is reported to rank first on the six native source–horizon cells and to improve exact-video-and-plan success by 2.9–10.7 points on four matched same-task COIN/CrossTask cells over an enhanced hard-selection SOTA. The paper also reports diagnostics on converted sources, stress tests on candidate count and feature noise, case analyses, and a supplementary code/constructors package.","tokens_in":16323,"tokens_out":7375,"duration_ms":82642,"significance":"If the empirical claims hold, CVSPP is a useful new benchmark that couples video corpus moment retrieval with procedure planning, and OSEF's delayed-commitment, token-global fusion design is a plausible architectural contribution. The release of model constructors and evaluation code, the typed-negative-role construction, and the attempted fail-closed audit are concrete strengths that go beyond many benchmark papers. However, two load-bearing issues temper the significance: the headline metric omits the localization component that the task definition itself requires, and the leakage audit is under-specified for a setting where queries and hidden answers are derived from the same annotation sources. The validation-selected reporting and the mixed-regime presentation of the hard-SOTA row also need addressing before the stated SOTA/rank claims can be taken at face value.","major_comments":[{"comment":"The task definition in Sec. 4 requires a prediction triple (k, i, j) — supporting video, start/end cell, and action sequence — and the Introduction says CVSPP 'must identify the supporting video, localize its window, and predict the intervening action sequence.' However, FULL-SR in Eq. (1) is defined as Pr[khat=k*, Ahat=A*] with no window term. The paper's own Limitations acknowledge that 'FULL-SR can be correct with a temporally wrong window,' and Sec. 6.3 reports CT-T3 FULL@IoU.3/IoU.5 of only 6.21%/0.54% while FULL-SR is 58.09%. The headline metric therefore does not measure the localization component that the paper advertises. This is not merely a naming issue: the claim 'first benchmark to score all three outputs together' (Sec. 1) is not supported by the stated metric. Please either add a window condition to FULL-SR (or a jointly thresholded variant) or explicitly rename the headli","section":"Sec. 4, Eq. (1); Sec. 6.3; Limitations"},{"comment":"The fail-closed leakage audit is under-specified. Sec. 3 states only that a normalized phrase audit compares every query against its target action labels, with matches human-adjudicated or quarantined. The legal queries are generated by deterministic templates from start–goal state descriptions, and those state descriptions and the hidden action sequences can come from the same annotation pipeline. An exact/substring phrase audit may not detect near-paraphrase leakage, so retrieval and planning scores in Tables 3–4 could in principle be inflated. The Sec. 6.6 controls (query-only FULL-SR 1.12%, task-name/wrong-transition R@1 changes of 1.81/0.66 points) are encouraging, but they cover only one hard same-task COIN cohort. Please provide a detailed audit protocol — including paraphrase/n-gram checks against captions, state descriptions, and action labels — and per-source query-only baselin","section":"Sec. 3, 'Legal query and negative roles'; Sec. 6.6"},{"comment":"Tables 3–4 report validation-selected performance, not locked-test evaluation: each metric comes from the earliest epoch with maximum validation FULL-SR. The paper acknowledges that the original run weights were not retained and that the tables therefore report validation rather than test numbers. A supplementary locked-split retrain is claimed to reproduce the four matched cells within 2.0 points, but this is not shown in the main paper. Since the central contribution is a ranking/SOTA comparison, the main paper should report locked-test numbers with retained weights, or at least include the locked-split table with per-cell values. Without that, the reported margins and ranks are not final.","section":"Sec. 6.1, 'Reporting protocol'"},{"comment":"The hard-selection SOTA row in Table 3 uses same-task candidate caches, while every other row (including OSEF) uses distinct-task candidate sets. Sec. 6.2 explicitly states this and says the row is 'not comparable' with the others, yet it is printed in the same table without visual separation, and it includes NIV entries even though the abstract's 2.9–10.7-point improvement is computed only on the four matched CT/COIN same-task cells in Table 4. As printed, the Table 3 layout invites the reader to compare, e.g., CT-T3 18.23 (hard SOTA, same-task) with 58.09 (OSEF, distinct-task), which are different regimes. Please move the hard-SOTA row into a clearly labeled same-task table (or remove it from Table 3), and state in the caption that Table 3's non-hard rows use distinct-task candidates.","section":"Table 3 and Sec. 6.2"}],"minor_comments":[{"comment":"The figure contains a typo: 'Ture✓' should be 'True✓'.","section":"Figure 1"},{"comment":"The symbol 'CET' is used without definition. Presumably it is a stepwise cross-entropy loss over the T action steps; please define it.","section":"Eq. (3)"},{"comment":"The footnote says 'Cells are seed 0/three-seed means' — this is ambiguous. Specify which cells are single-seed and which are three-seed means, and give the seed SDs in the main table or a companion table.","section":"Table 3 footnote"},{"comment":"FULL@IoU.3 and FULL@IoU.5 are used in Sec. 6.3 without formal definitions in the metric contract. Please state the exact conjunction (video = k*, plan = A*, and window IoU above the threshold) in Sec. 4.","section":"Sec. 4, metric contract"},{"comment":"The statement that the best adapted route exceeds the majority floor on ANet and CEgo 'by only 3 and 6 strict successes' is hard to interpret without per-cell sample sizes. Please report the numbers in the form 'X/Y examples above the floor' or provide the exact counts in the per-cell supplement.","section":"Sec. 6.3"},{"comment":"The text says 'Eight more sources' are converted/diagnostic, but Table 2 lists nine non-native entries (YC2, TACoS, HiR, GS, ANet, CEgo, GUIDE, ELearn, plus diagnostic EEL). Clarify whether EEL is included in the 'eight' or counted separately.","section":"Sec. 3, source list"}],"recommendation":"major_revision","confidential_remarks":"The paper is a substantial benchmark-and-method submission with a real architectural idea, and the supplementary code release is a strength. The main blockers are fixable but load-bearing: the headline metric does not match the task's localization requirement, the leakage audit needs to be made convincingly fail-closed across all sources, and the main tables should report locked-test numbers. I would not reject, but I would not accept without these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real new task and a serious benchmark-plus-method paper, not a rehash. The new thing is CVSPP—retrieve a video, localize a window, and predict a closed-set plan from one answer-redacted query—and OSEF, which delays hard selection by scoring a query-conditioned lattice over all candidates and feeding the whole soft lattice to the planner through a token-global adapter. The benchmark construction is thoughtful: typed negatives, eleven sources, a majority-sequence floor, and an explicit decision to only rank on the six native cells. The component study is unusually clear: the token-global interface, not soft pooling or the extra losses, drives the gain. That is a real result, and the paper is honest about its own limits.\n\nThe soft spots are real but proportionate. Eq. (1) defines FULL-SR without a window term, so the headline metric does not actually require localization even though the task promises it; the paper acknowledges this and reports FULL@IoU separately, but those numbers are very low (e.g., CT-T3 6.21%/0.54%), which weakens the “localize” part of the story. Tables 3–4 report validation-selected checkpoints rather than locked-test numbers; the authors disclose this and offer a prospectively specified locked-split retrain for four matched cells, which is better than common practice but still not a clean held-out evaluation.\n\nThe leakage question is the one I would push hardest. The fail-closed audit described in Sec. 3 checks normalized phrase matches against target action labels, but the queries are generated from deterministic templates over state descriptions, and those descriptions may paraphrase the hidden action sequence. The query-only and wrong-transition controls are encouraging—FULL-SR drops from 18.70% to 1.12% query-only, and R@1 moves only 1.81 points—but they cover one hard same-task COIN cohort, not all fourteen cells, and they do not test a strong text-only baseline. I would want the audit tightened or an external paraphrase-probe before trusting the absolute retrieval and plan scores. The relative claim, however, is matched and survives the leakage worry: under the same frozen evidence, OSEF beats hard-selection on four same-task COIN/CrossTask cells by 2.9–10.7 points, and the ablation assigns the gain to the interface.\n\nCiting and literature handling looked sane; the related work is well positioned and the paper does not overclaim the converted-source cells. Who it is for: people working on instructional video planning, retrieval-to-plan pipelines, or video agents that plan from search results. It deserves a serious referee—the task definition and benchmark design are important enough, and the authors are transparent about limitations. Send it out, with the leakage audit as the main review ask.","headline":"A genuinely new benchmark and a clean interface ablation, but the window-less headline metric and the under-specified leakage audit keep me from trusting the absolute numbers as much as the relative claim.","tokens_in":16804,"tokens_out":2140,"would_cite":true,"duration_ms":27420,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces Cross-Video Scene Procedure Planning (CVSPP) and OSEF, which fuses all candidate evidence softly and improves exact-video-and-plan success by up to 10.7 points on same-task COIN and CrossTask cells.","keywords":["CVSPP","procedure planning","video retrieval","temporal localization","evidence fusion","soft lattice","instructional videos","retrieval-to-plan error propagation"],"falsifier":"A direct audit: insert a detectable cue into the legal query (e.g., the first action verb) and re-run the full benchmark; if FULL-SR does not collapse, the redaction gate is leaking. Alternatively, compute FULL-SR when the evidence branch is given the true video id as an additional input; if OSEF's margin over hard selection shrinks to zero, the gain is entirely in retrieval, not in plan fusion.","tokens_in":15860,"feed_emoji":"🎬","tokens_out":7198,"duration_ms":65252,"temperature":0.7,"pith_summary":"The paper establishes a new task, Cross-Video Scene Procedure Planning (CVSPP): given a start–goal query that has been redacted so it cannot leak the answer, and a small set of candidate videos, a model must find the right video, locate the right segment, and output the exact action sequence. The authors argue that two coupled failures—same-task evidence ambiguity and retrieval-to-plan error propagation—make retrieve-then-plan cascades brittle. Their method, One-Step Evidence Fusion (OSEF), scores every cell of every candidate against the query and feeds the whole soft lattice to the planner through a token-global adapter, never cropping a window first. On the benchmark's six method-rankable cells OSEF ranks first, and on four matched same-task cells it improves exact-video-and-plan success by 2.9–10.7 percentage points over the strongest hard-selection baseline. A component study attributes the largest single gain to the token-global interface, not to soft pooling.","feed_headline":"Soft evidence lattice beats hard selection by up to 10.7 points","feed_subtitle":"Fusing all candidates into one soft lattice lifts exact-video-and-plan success by up to 10.7 points.","key_machinery":"The token-global Evidence-to-Plan (E2P) adapter: a one-layer Transformer that contextualizes the flattened K×M cell-feature lattice (with a learned score projection injecting evidence logits), followed by 2T learned endpoint queries that cross-attend to all cells to produce T start–goal state pairs. This is what lets the planner retain every candidate's evidence; the query-conditioned cell-and-span lattice Hθ(q,k,i,j) = ½(Lθ(q,k,i)+Lθ(q,k,j)) supplies the evidence scores, and a joint-cascade loss couples plan error to candidate probabilities.","core_discovery":"The central discovery is that hard evidence selection—choosing one video and one window before planning—is the main bottleneck in cross-video procedure planning, and that keeping evidence soft through the planner is a better strategy. OSEF computes a query-conditioned lattice of cell scores, lifts them to valid intervals via a symmetric formula, and passes the entire lattice into a goal-conditioned Transformer planner through a token-global adapter. The paper reports that this one-step fusion path improves FULL-SR (exact video and exact action sequence) by 2.9–10.7 points over an enhanced hard-selection SOTA on four matched COIN/CrossTask cells, and that the token-global interface—rather tha","pith_inferences":["Inference: If the token-global adapter is the key, then a pure soft-pooling or hard-crop variant should lose the same 3–6 points on any new same-task benchmark, not just COIN/CT.","Inference: The fail-closed leakage gate is a strong protocol innovation; other retrieval-planning benchmarks could adopt the same redaction audit to prevent answer leakage.","Inference: The near-zero FULL@IoU.5 numbers suggest that OSEF's headline FULL-SR metric may overstate practical usability for applications that need the exact segment; a window-aware variant would be the next logical benchmark cell.","Inference: Because K=4 and M=16 are fixed, the paper's K-degradation curve (R@1 29.48→8.12 as K goes 4→16) indicates that same-task retrieval, not planning, will be the scalability bottleneck for larger candidate sets."],"forward_implications":["On the six benchmark cells that support method ranking, OSEF is first, and on the four matched same-task COIN/CrossTask cells it beats the enhanced hard-selection SOTA by 2.9–10.7 FULL-SR points.","The token-global interface, not soft pooling or the coupled losses, is the main source of the gain (Table 4).","Fusion never falls below the hard baseline on any row, and the full stack lifts COIN-T3 PLAN-SR from 46.61% to 55.34%.","Distinct-task retrieval is nearly saturated on native sources (NIV-T3 R@1 99.20%), leaving exact plan composition as the bottleneck; T=4 lowers FULL-SR by 17.81–22.70 points while R@1 changes by less than a point.","Localization remains weak: CT-T3 FULL-SR 58.09% drops to 6.21% FULL@IoU.3, so a correct plan does not certify the window."],"fun_headline_variants":["Soft evidence fusion beats hard selection by up to 10.7","One-step soft lattice tops hard selection by up to 10.7","OSEF: Fusing all video evidence lifts exact-video-and-plan","Hard selection is bottleneck: soft evidence wins by up to 10.7","Soft evidence beats hard selection in cross-video planning"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The redacted start–goal query must be genuinely uninformative about the hidden video, window, and action sequence; if the fail-closed audit misses a leak, every retrieval and planning score could be inflated without the model solving the intended task.","fun_headline_variants_meta":{"raw":{"variants":["Soft evidence fusion beats hard selection by up to 10.7","One-step soft lattice tops hard selection by up to 10.7","OSEF: Fusing all video evidence lifts exact-video-and-plan","Hard selection is bottleneck: soft evidence wins by up to 10.7","Soft evidence beats hard selection in cross-video planning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1215,"prompt_tokens":814,"completion_tokens":401,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":309}},"tokens_in":558,"tokens_out":401,"duration_ms":6001,"temperature":1.0,"reasoning_tokens":309,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T07:44:16.072124+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct audit: insert a detectable cue into the legal query (e.g., the first action verb) and re-run the full benchmark; if FULL-SR does not collapse, the redaction gate is leaking. Alternatively, compute FULL-SR when the evidence branch is given the true video id as an additional input; if OSEF's margin over hard selection shrinks to zero, the gain is entirely in retrieval, not in plan fusion.","supporting_citations":[],"review_version":1}