{"id":"0e7af769-be6b-4999-a07a-400b9a0d56a7","arxiv_id":"2608.12780","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SCOPE accelerates video diffusion transformers about 2x by scoring keys through 3D-RoPE subspace clusters and adaptively setting per-head Top-k counts, matching dense-attention fidelity closely.","lead":"SCOPE speeds up video generation from text or images by skipping most attention computations, while keeping the output close to the full-attention version. It groups video tokens by position and estimates how many tokens each attention head needs, cutting inference time by up to about 2x with minimal quality loss.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (6) proxy ranking is the load-bearing component of the fidelity claim, yet it is validated on only one query token and is not auditable because the Top-p threshold ρ is never reported.","rationale":"The paper's novelty is a cheap proxy that preserves key-level distinctions (Eq. 6) plus an adaptive count floor (Eq. 9). If the proxy ranking were poor, both the Top-p counts and k_head would be computed over the wrong ordering, so SCOPE's masks would drop high-contribution keys and the reported PSNR gains would vanish. This matches the reader's weakest assumption. The paper provides one supportive visualization (Fig. 8) and an ablation (Table 3) favoring the 3D-RoPE partition, but neither spans the layer/head/step space nor reports recall statistics, and the missing ρ makes the failure mode impossible to audit externally. I would not reject on this basis: the additive decomposition in §3 is internally consistent, the design choices are ablated, and the proposed validation is straightforward to run. I therefore keep the reader's CONDITIONAL verdict. One additional observation is that in Table 1 the HunyuanVideo-T2V latency gap over SVG2 is 3 s out of roughly 905 s, so the 'lowest latency' half of the headline claim also needs repeated-run evidence; this supports, rather than replaces, the primary concern about proxy-ranking fidelity.","tokens_in":16275,"tokens_out":8789,"duration_ms":96749,"concrete_test":"Re-run SCOPE on HunyuanVideo-T2V with dense attention logits cached; for a stratified sample of layers/heads/denoising steps, compute the oracle recall of SCOPE's selected top-r_c key sets and the rank correlation between Eq. (6) proxy logits and true logits, then sweep the unreported Top-p threshold ρ ∈ {0.5, 0.7, 0.9, 0.95, 0.99} at fixed α=0.1 and report PSNR, realized density, and latency with at least three repetitions per setting. If oracle recall is not consistently high or the PSNR/latency trade-off shifts materially with ρ, the central claim fails; the repetitions also determine whether the 904s vs 907s Hunyuan latency gap over SVG2 is meaningful.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central fidelity claim depends entirely on the proxy logit \\tilde{s}_{c,j} in Eq. (6) ranking high-contribution keys nearly as accurately as true attention scores, so that the hybrid Top-p/Top-k selection and the online per-head k_head in Eqs. (7)-(10) retain the right keys. This assumption is not established. Figure 8 analyzes a single representative query token at one layer/head/denoising step; it does not quantify recall or ranking fidelity across the six configurations, 129-frame videos, or the many heads and denoising steps where new query and key centroids are computed. There is also no analytical guarantee: each subspace centroid is chosen by k-means to minimize Euclidean reconstruction error in Eq. (3), not the dot-product error that determines attention ordering, so neither §3 nor Appendix B provides a bound showing Eq. (6) preserves the oracle ranking. The omission of the Top-p threshold ρ is compounding: Eq. (7) is the foundation of the online estimator, but the implementation details report only Cq=300, Cm=333, and α=0.1. Without ρ the reported PSNR and latency numbers cannot be reproduced and may reflect a tuned operating point. The load-bearing assumption is therefore currently untestable from the manuscript as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SCOPE, a training-free sparse-attention method for video diffusion transformers. It partitions post-RoPE keys into temporal, height, and width channel subspaces, clusters each subspace independently, and scores each key through lookup tables of query-centroid/key-centroid dot products. A hybrid Top-p/fixed-Top-k selection per query cluster is followed by an online per-head Top-k estimate obtained as the query-size-weighted average of base retained counts. Sparse attention is then computed with the original selected keys and values. Experiments on six 720p text-to-video and image-to-video settings across Wan2.1, Wan2.2, and HunyuanVideo report higher PSNR/SSIM/LPIPS and lower latency than SpargeAttn, SVG2, and SVOO, with 1.67x-1.99x end-to-end speedups.","tokens_in":16551,"tokens_out":8062,"duration_ms":73106,"significance":"If validated, this is a useful contribution to training-free sparse attention for video DiTs. The compositional scoring identity in Eq. (27) is exact, so the key-side approximation is cleanly characterized as replacing each key by its product-code representative; the complexity analysis in Appendix B separates the additive scoring cost from the multiplicative code space. The online per-head Top-k is a simple and input-adaptive heuristic whose benefit is supported by the matched-density ablation in Figure 7. The evaluation is broad and consistent: six configurations across three model families, with matched-density evidence in Figure 2 and no sign of circularity, since fidelity is measured against dense attention and latency against end-to-end runtime. The main weakness is that the ranking fidelity of the proxy logit, which is the load-bearing component of the fidelity claim, is validated on only one representative query token, and key configuration details (the Top-p threshold and whether the dense-prefix schedule is shared by baselines) are not reported. These gaps make the central claim currently underevidenced, though all are fixable within the manuscript's scope.","major_comments":[{"comment":"The central fidelity claim rests on the proxy logit of Eq. (6) ranking keys nearly as accurately as true attention scores, but the manuscript does not establish this. Eq. (27) shows that the proxy logit is exactly the inner product with the product-code representative, so the approximation error is <\\bar q_c, k_j - \\hat k_j>/sqrt(d); however, the centroids in Eq. (3) are chosen by k-means to minimize Euclidean reconstruction error per subspace, which is not the same objective as preserving the dot-product ordering that determines attention. Figure 8 analyzes a single representative query token at one layer/head/denoising step and reports mask granularity at matched oracle recall; it does not quantify how often high-contribution keys are mis-ranked across the six configurations, 129-frame sequences, heads, and denoising steps. Please provide either a bound on the induced ranking error or systematic recall/ranking-fidelity statistics over heads, layers, and denoising steps; without this, the fidelity gains in Tables 1-2 cannot be attributed to the proxy's ranking quality.","section":"§3, Eq. (6); Appendix C, Figure 8"},{"comment":"The Top-p threshold ρ used in Eq. (7) is never reported. The implementation details specify Cq=300, CT=CH=CW=333, α=0.1, and the dense-prefix schedule, but not ρ; Figure 7 states that p was set separately for each variant to match realized attention density, again without giving values. Because the base counts b_c in Eq. (8) and hence the online k_head in Eq. (9) and the final r_c in Eq. (10) all depend directly on ρ, the reported PSNR and latency numbers cannot be reproduced, and it is unclear whether the operating point is a single fixed ρ or a per-model tuned value. Please report ρ for each of the six settings (or state that it is fixed) and report the realized attention density per method.","section":"§4, Implementation details; Eq. (7)"},{"comment":"The dense-prefix schedule (first attention layer dense at every denoising step; dense for the first 20% of steps on Wan2.1/Wan2.2 and 10% on HunyuanVideo) is described only for SCOPE. The baselines are stated to use official implementations and default configurations, so it is unclear whether SpargeAttn, SVG2, and SVOO receive the same dense-prefix treatment. If they do not, the fidelity advantage in Tables 1-2 may be partly attributable to the dense prefix rather than to subspace clustering and online Top-k. Please state explicitly whether the dense-prefix schedule applies uniformly to all methods, or add a comparison in which all methods use the same dense-prefix schedule.","section":"§4, Implementation details; Tables 1-2"},{"comment":"The claim that SCOPE uses the same sparse-attention budget more effectively is supported by Figure 2, but Tables 1 and 2 do not report attention density for each method. Since latency differences across methods are small in some settings (for example, 904s for SCOPE versus 907s for SVG2 in Table 1, HunyuanVideo T2V), a reader cannot tell whether the PSNR advantage comes from better key selection or from a higher retained-key count at the reported operating points. Please include retained-density values in the main tables or in a companion table, and state how the operating points were matched.","section":"Tables 1-2 and Figure 2"}],"minor_comments":[{"comment":"The abstract reports 'up to a 1.99x end-to-end speedup on 720p HunyuanVideo with 28.46 dB PSNR', while Figure 1 shows a per-run HunyuanVideo result of 2.00x speedup and 29.15 dB PSNR. Please clarify whether the abstract reports an average or maximum over runs and reconcile the 2.00x value.","section":"Abstract and Figure 1"},{"comment":"The Figure 4 caption states Cvis = 10, whereas the implementation details use CT = CH = CW = 333. Please explain that the figure uses a small visualization codebook and clarify the relationship to the actual configuration.","section":"Figure 4"},{"comment":"The caption contains a typo: 'set p seperately' should read 'set p separately'.","section":"Figure 7"},{"comment":"The complexity analysis in Appendix B excludes the cost of the K-means clustering itself, covering only the table-construction and lookup stages after codebooks and assignments are available. Since Figure 6 reports clustering overhead as small, consider adding a short complexity statement for the clustering stage, or a note explaining why it is amortized or small.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The core idea is sound and the evaluation is broad, but the load-bearing proxy-ranking assumption needs substantially more evidence than a single-token visualization, and the missing Top-p threshold and unclarified dense-prefix sharing are reproducibility and fairness issues. These are fixable within the manuscript's scope. I would be comfortable with acceptance after a revision that adds systematic proxy-ranking validation and full configuration reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nSCOPE is one of the more careful training-free sparse attention papers I've seen for video DiTs. It beats three strong baselines (SpargeAttn, SVG2, SVOO) on fidelity and latency across all six 720p T2V/I2V settings, and the core mechanism is genuinely new: partition post-RoPE keys along temporal/height/width channel ranges, cluster each subspace independently, and compose per-subspace centroid scores into a per-key proxy logit. Eq. (27) is a clean identity—the lookup-and-sum is exactly the inner product with the product-code representative, so the only approximation is the k-means substitution. The online per-head Top-k (Eq. 9) is a sensible, cheap fix for under-selection, and the ablation in Fig. 7 shows it earns real PSNR at matched density.\n\nThe soft spots are real but not disqualifying. The load-bearing claim is that the proxy logit ranks high-contribution keys nearly as well as true attention, and the paper validates that on a single representative query token (Fig. 8). No recall/ranking statistics across layers, heads, denoising steps, or videos. Second, the Top-p threshold ρ is never reported—it appears in Eq. (7) and Algorithms 1/3 but not in the implementation details—so the reported PSNR/latency numbers aren't reproducible from the manuscript alone. No code release, no error bars, and hyperparameters (Cq=300, Cm=333, α=0.1, dense-prefix fraction) look like they were chosen on these benchmarks without sensitivity analysis. These are reporting gaps, not signs of a broken method; nothing in the paper contradicts its own claims.\n\nWho's it for: researchers working on efficient video diffusion inference. It deserves a serious referee and a request for the missing details. I'd bring it to reading group and cite it as related work, but I wouldn't yet trust the exact fidelity numbers until ρ is disclosed and the proxy-ranking validation is broadened.\n\nSend to peer review. Conditional accept, with the authors asked to report ρ, release code, and add a quantitative evaluation of proxy ranking fidelity across the full head/layer/step sweep.","headline":"A legitimate, well-engineered sparse-attention method with a genuinely novel subspace-composition proxy; the core ranking assumption is under-validated and a missing top-p threshold blocks reproduction, but it deserves serious review.","tokens_in":17144,"tokens_out":3250,"would_cite":true,"duration_ms":30833,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SCOPE claims that a training-free sparse attention method can make video diffusion transformers run up to 1.99x faster end-to-end while staying closer to the dense-attention reference than existing training-free baselines.","keywords":["sparse attention","video diffusion transformers","training-free acceleration","subspace clustering","3D rotary position embedding","per-head Top-k estimation","attention sparsity"],"falsifier":"Compare the keys SCOPE retains against the oracle top-10% keys from dense attention on a held-out set of prompts. If, at matched oracle recall, the retained fraction is not substantially below full-dimensional cluster scoring, or if reducing the subspace codebook size to roughly 200 causes recall to collapse, the granularity claim fails. A second check is to run SCOPE on a video DiT without 3D RoPE or with permuted channel order; if the gains disappear, the method depends on that exact partition.","tokens_in":16066,"feed_emoji":"🎬","tokens_out":4664,"duration_ms":42759,"temperature":0.7,"pith_summary":"The paper is trying to establish that a training-free sparse attention method can accelerate video diffusion transformers while preserving dense-attention fidelity better than existing training-free methods. It argues that two coupled fixes are needed: finer-grained key scoring via 3D-RoPE-aligned subspace clustering, and input-adaptive retained-key counts via online per-head Top-k estimation. If the claim is right, sparse video attention does not require retraining or offline dense calibration to be both fast and faithful. The evidence is six 720p text-to-video and image-to-video configurations across three video DiT families, with up to a 1.99x end-to-end speedup and 28.46 dB PSNR relative to dense attention on HunyuanVideo T2V.","feed_headline":"SCOPE speeds up video attention 1.99x without retraining","feed_subtitle":"Training-free sparse attention with per-head Top-k keeps dense-attention fidelity at lower latency.","key_machinery":"The central object is the compositional proxy logit $\\tilde{s}_{c,j} = \\frac{1}{\\sqrt{d}} \\sum_{m \\in \\{T,H,W\\}} P^m[c, z^m_j]$, where each key's temporal, height, and width subspace assignment looks up one centroid dot product per subspace and the three are summed. This gives every key a distinct estimated importance at additive scoring cost while the product of the subspace codebooks provides a multiplicative representation space. The second mechanism is the online per-head Top-k estimate $k_{\\text{head}} = \\left\\lceil \\frac{\\sum_c n_c b_c}{\\sum_c n_c} \\right\\rceil$, which extends only query clusters whose initial hybrid Top-p/fixed Top-k count falls below the head-level average. The first mechanism recovers fine-grained key discrimination; the second makes the retention floor adapt per head and per input without offline calibration.","core_discovery":"On the paper's own terms, the central discovery is that post-RoPE keys in video DiTs can be split along the temporal, height, and width channel ranges of 3D rotary position embeddings, clustered independently, and scored by summing centroid lookup-table entries, producing per-key proxy logits that rank keys much more finely than block-level or full-dimensional cluster-level proxies. The paper then adds a head-specific Top-k floor computed online as the query-size-weighted average of the initial hybrid Top-p/fixed Top-k retained counts, which corrects under-selection without offline dense profiling. Together these mechanisms yield the best measured fidelity and lowest measured latency among the evaluated training-free baselines, with the reported speedups of 1.67x to 1.99x and PSNR gains over baselines in every setting.","pith_inferences":["A testable extension is to apply the same subspace-composition scoring to other RoPE-based transformers outside video DiTs, such as long-context LLMs, where fine-grained per-key ranking could improve sparse attention or KV-cache eviction without training.","The online per-head Top-k estimator could be reused as a calibration-free fallback in any hybrid Top-p/Top-k selector, potentially combined with trainable policies that adjust the base counts instead of only the floor.","Because the paper reports overhead as a small share of end-to-end latency, another extension is to reuse or update subspace codebooks across adjacent denoising steps, which would cut K-means cost further while likely preserving most of the fidelity gain.","The claim that 3D-RoPE-aligned partitions outperform random partitions suggests a testable hypothesis: the gains should shrink on models whose RoPE channel ordering is rearranged, and grow when temporal, height, and width structures are more distinct."],"forward_implications":["If SCOPE is correct, the same sparse attention budget can be used more effectively: at matched attention density it achieves higher attention recall and PSNR than block-level and cluster-level proxy baselines.","Sparse attention can be applied to existing video DiTs without retraining or modifying model parameters, since all proxy quantities are used only for key selection and attention is computed with the original tokens.","The online per-head Top-k estimator removes the need for offline dense-attention profiling or stored head-wise schedules, so the method adapts to different heads and inputs at inference time.","The measured end-to-end speedups of 1.67x to 1.99x imply that the overhead of subspace clustering, proxy scoring, and online selection is consistently outweighed by the reduction in attention computation.","The 3D-RoPE-aligned partition is claimed to outperform random channel partitions and full-dimensional clustering at the same centroid budget, indicating that the alignment with the rope channel structure is load-bearing for fidelity."],"supporting_citations":[{"why":"Supplies the product-quantization principle of decomposing vectors into independently clustered subspaces with additive scoring.","marker":"[11]"},{"why":"Defines the HunyuanVideo model family and its T2V benchmark prompts evaluated in the experiments.","marker":"[13]"},{"why":"The SVOO baseline, which relies on offline sparsity profiling and online QK co-clustering; SCOPE claims no offline dense calibration is needed.","marker":"[23]"},{"why":"Introduces rotary position embeddings, the basis of the 3D-RoPE channel ranges SCOPE aligns its key subspaces with.","marker":"[31]"},{"why":"Defines the Wan model family used for the Wan2.1 and Wan2.2 evaluation configurations.","marker":"[34]"},{"why":"Provides the video-specific 3D rotary position embedding whose temporal, height, and width channel ranges SCOPE uses as the key partition.","marker":"[36]"},{"why":"The SVG2 baseline, which supplies the semantic-aware permutation approach and the K-means implementation reused by SCOPE.","marker":"[41]"},{"why":"The SpargeAttention baseline, a block-wise proxy scoring method that SCOPE compares against and improves on.","marker":"[46]"},{"why":"Introduces hybrid Top-p/Top-k selection in a trainable setting, which SCOPE adapts into an online per-head estimation.","marker":"[47]"},{"why":"Provides the VBench quality metrics used to report generation quality alongside dense-reference fidelity.","marker":"[9]"}],"fun_headline_variants":["SCOPE: 1.99x faster video attention without training","Training-free sparse attention: 1.99x video speedup","Per-head top-k clusters keys for 1.99x video-DiT speed","SCOPE: subspace clustering speeds video diffusion 1.99x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proxy logit obtained by summing per-subspace centroid dot products ranks keys in nearly the same order as true attention scores across all layers, heads, and denoising steps, which requires the 3D-RoPE channel ranges to segment keys into subspaces whose independent clustering preserves the distinctions that determine high-contribution keys.","fun_headline_variants_meta":{"raw":{"variants":["SCOPE: 1.99x faster video attention without training","Training-free sparse attention: 1.99x video speedup","Per-head top-k clusters keys for 1.99x video-DiT speed","SCOPE: subspace clustering speeds video diffusion 1.99x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001439,"raw_usage":{"total_tokens":5837,"prompt_tokens":1019,"completion_tokens":4818,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":4737}},"tokens_in":635,"tokens_out":4818,"duration_ms":32466,"temperature":1.0,"reasoning_tokens":4737,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:24:58.957604+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the keys SCOPE retains against the oracle top-10% keys from dense attention on a held-out set of prompts. If, at matched oracle recall, the retained fraction is not substantially below full-dimensional cluster scoring, or if reducing the subspace codebook size to roughly 200 causes recall to collapse, the granularity claim fails. A second check is to run SCOPE on a video DiT without 3D RoPE or with permuted channel order; if the gains disappear, the method depends on that exact partition.","supporting_citations":[{"cited_title":"Sparse videogen2: Accelerate video generation with sparse attention via semantic-aware permutation.Proc","cited_arxiv_id":null,"evidence_quote":"The SVG2 baseline, which supplies the semantic-aware permutation approach and the K-means implementation reused by SCOPE."},{"cited_title":"Spargeattention: Accurate and training-free sparse attention accelerating any model inference","cited_arxiv_id":null,"evidence_quote":"The SpargeAttention baseline, a block-wise proxy scoring method that SCOPE compares against and improves on."}],"review_version":1}