{"id":"ef20a54b-3020-447a-98bb-0fc210f98632","arxiv_id":"2502.01776","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Sparse VideoGen accelerates video diffusion transformers by about 2.3x with only small quality loss by classifying attention heads into spatial and temporal sparse patterns and using hardware-friendly layouts.","lead":"This paper introduces Sparse VideoGen, a training-free method that speeds up video generation models by skipping most of the attention computation. It classifies attention heads as either spatial or temporal and uses specialized sparse kernels to cut runtime by about 2.3x while keeping output close to the original.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 1% query sampling that decides each head's spatial/temporal mask is the load-bearing assumption; misclassification of rare-but-important tokens could break the 'preserving quality' claim, which Table 3 does not rule out.","rationale":"The reader's weakest assumption is exactly the point I would stress. The online profiler in Algorithm 1 is the only mechanism that decides whether a head is computed as spatial or temporal, and the decision is made from 1% of query rows per head per step. If that sample misrepresents the head, the error is not local: the chosen mask is applied to the entire head, and since no fallback or verification exists, a few misclassified heads per step can compound over the many denoising steps and transformer layers. The paper's Table 3 shows that aggregate PSNR with 1% profiling is close to the 100% oracle (31.118 vs 31.324 on one CogVideoX subset), which is reassuring, but it does not measure per-head decision agreement, nor does it report variance across prompt categories. Randomly sampled VBench prompts are unlikely to contain the adversarial cases, such as fast motion, rare objects, or multiple interacting subjects, where a minority temporal pattern in a mostly-spatial head matters most. A concrete per-head/step agreement test would settle this. I also note, as a secondary point, that the headline 2.33x speedup on HunyuanVideo includes FP8 quantization and custom QK-norm/RoPE kernels; Figure 7 attributes 1.81x to sparse attention itself. The full-system speedup is real and reproducible, but the paper should be careful not to let 'sparsity' claim credit for the entire 2.33x. This does not change the conditional verdict: the method is credible, open-sourced, and the remaining issues are qualifications and missing measurements, not demonstrated failures.","tokens_in":17141,"tokens_out":9705,"duration_ms":93916,"concrete_test":"Use the released implementation to log per-head/step decisions under 1% profiling and under 100% oracle profiling on a stratified prompt set (VBench categories with fast motion, rare objects, multiple subjects, and scene cuts). Compute per-head decision agreement and per-category PSNR/LPIPS of 1%-profiled SVG versus dense output. If any category shows decision agreement below roughly 95% or a PSNR drop greater than 1 dB relative to the 100%-profiled oracle, the 'preserving generation quality' claim must be qualified to typical prompts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 / Algorithm 1 selects, for every head and denoising step, either the spatial or temporal mask by comparing MSE against full attention on only 1% randomly sampled query rows; the chosen mask is then applied to all queries in that head. This is a proxy for the head's true best pattern, and it fails exactly when a head is spatial for most tokens but temporal for a small, semantically critical subset (e.g., a fast-moving object or a rare interaction). Those critical tokens are unlikely to be in the 1% sample, so the head is classified by its majority behavior and the minority pattern is computed incorrectly for the whole head. Because the method has no fallback and errors can accumulate over roughly 50 denoising steps and many layers, aggregate PSNR on a random VBench subset (Table 3) is not enough to establish 'preserving generation quality': the table reports only video-level quality at a few profiling ratios, not per-head/step classification agreement with the 100% oracle, and it does not stratify by motion or prompt difficulty. The central claim therefore rests on an unverified representativeness assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Sparse VideoGen (SVG), a training-free framework for accelerating video diffusion transformers by exploiting sparsity in 3D full attention. It classifies attention heads into spatial and temporal types via an online profiling strategy that samples 1% of query rows, uses a hardware-efficient layout transformation to make temporal-head sparsity tensor-core friendly, and adds customized kernels and optional FP8 quantization. On CogVideoX-v1.5-I2V, CogVideoX-v1.5-T2V, and HunyuanVideo-T2V, the authors report end-to-end speedups of up to 2.28x and 2.33x (with FP8) while maintaining PSNR values around 28-30, outperforming baselines such as DiTFastAttn, MInference, and PAB. The code is open-sourced.","tokens_in":17420,"tokens_out":13575,"duration_ms":109243,"significance":"If the reported results hold, SVG is a practical, training-free acceleration method for video DiTs, with the key novelty being the online per-head classification into spatial and temporal patterns and the layout transformation that turns a hardware-hostile strided pattern into a contiguous block-sparse one. The paper's strengths include end-to-end wall-clock measurements on H100 across multiple models, a clear ablation of the layout transformation (Figure 8) showing that the system achieves near-theoretical speedups, and open-sourced code that supports reproducibility. The main uncertainty is whether the 1% profiling assumption is robust across diverse prompts and motions, which the current aggregate metrics do not yet establish.","major_comments":[{"comment":"The central quality-preservation claim rests on the assumption that per-head masks selected from 1% of randomly sampled query rows match the oracle (100%) selection for every head, denoising step, and prompt. Table 3 provides only aggregate PSNR/SSIM/LPIPS on a random subset of VBench for CogVideoX-I2V, which does not rule out misclassification of rare-but-important tokens (e.g., fast-moving objects or rare interactions) that are unlikely to appear in the 1% sample. Since there is no fallback to dense attention when the sampled MSE is ambiguous, and errors can accumulate over the roughly 50 denoising steps, the claim that SVG 'preserves generation quality' is not fully established. Please add per-head/step mask-agreement rates against the 100%-profiling oracle, or stratify the evaluation by motion/prompt difficulty, or report worst-case (e.g., 5th percentile) PSNR across the benchmark.","section":"Section 4.1, Algorithm 1, Table 3"},{"comment":"The quality metrics that support the 'preserving generation quality' claim are reported as point estimates without error bars, confidence intervals, or significance tests. For example, Table 3 shows a difference of 0.206 PSNR between the 1% and 100% profiling ratios; without the number of prompts, random seed, or run-to-run variance, it is unclear whether this difference is meaningful or within noise. Please report the number of prompts, the random seed, and error bars (or per-prompt distributions) for the core quality metrics.","section":"Section 5.2, Tables 1, 3, 4"},{"comment":"The temporal-head pattern is described inconsistently with the configuration parameters. Section 3.1 states that a temporal head attends to tokens at the same spatial position across frames (a slash-wise pattern with stride L), which would involve about N tokens per query (e.g., 11 for CogVideoX), but Section 5.1 sets ct = 1224 for CogVideoX and ct = 1200 for HunyuanVideo, orders of magnitude larger than N. The exact definitions of gen_spatial_mask() and gen_temporal_mask() in Algorithm 1 are never given, and this ambiguity propagates to the theoretical analysis in Section 3.3. Please provide the precise mask construction and reconcile the narrative with the parameter choices.","section":"Section 3.1, Section 5.1, Algorithm 1"},{"comment":"The paper uses 'sparsity' to mean the fraction of full attention computation retained (i.e., density), which is the opposite of the standard usage in the sparse-attention literature, where sparsity denotes the fraction of tokens skipped. For example, Section 3.3 says 'the sparsity can easily achieve 30%' while the configuration computes about 30% of the full-attention FLOPs (so 70% is skipped). This inverted terminology makes Table 4 and Figure 8 difficult to interpret (e.g., LPIPS appears to improve as 'sparsity' increases). Please rename the quantity to 'density' or 'computation ratio' throughout and update the text, table, and figure axes accordingly.","section":"Section 3.3, Table 4, Figure 8"}],"minor_comments":[{"comment":"The abstract reports a 1.51x speedup on Wan 2.1, but no quantitative Wan 2.1 results appear in the experiments section; the appendix only shows visual comparisons. Please add the quantitative benchmark or remove the claim from the abstract.","section":"Abstract / Section 5"},{"comment":"The title 'Sparse attention achieves lossless accuracy' overstates the evidence: the reported PSNR values (28-30) and the approximation in Table 3 are not strictly lossless. Suggest rewording to 'near-lossless' or 'high-fidelity'.","section":"Section 3.2"},{"comment":"The comment 't: Sampled Token Number. E.g., 32' is inconsistent with the 1% profiling ratio used in the experiments (which would be roughly 448 tokens for CogVideoX). Please clarify how t is set in practice.","section":"Algorithm 1"},{"comment":"The statement that profiling adds only 3% runtime overhead is not backed by a table or figure; please provide the measurement or point to an appendix entry.","section":"Section 4.1"},{"comment":"The speedup annotations in Figure 8 overlap and are difficult to read; please reposition them for clarity.","section":"Figure 8"},{"comment":"The phrase 'We skip the first 25% denoising steps for all baselines' is ambiguous about whether the dense baseline is also affected; please clarify that this applies to the accelerated baselines only.","section":"Section 5.1"},{"comment":"There are minor typos: 'instinct sparsity' in the Section 3.1 heading should be 'intrinsic sparsity', and 'Comparion' in the Appendix B captions should be 'Comparison'.","section":"Section 3.1 and Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems contribution with a clear speedup result, but the quality-preservation claim needs stronger validation in revision. The 'sparsity' terminology should be corrected before publication to avoid confusion with the standard usage. The Wan 2.1 speedup claimed in the abstract should either be supported with quantitative results or removed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a solid systems paper that delivers what it claims. The speedups are measured end-to-end on H100 (2.28x on CogVideoX-v1.5, 2.33x on HunyuanVideo), the code is open-source, and the kernel-level ablation in Figure 8 shows the layout transformation is what turns theoretical sparsity into actual speedup. That combination of honest wall-clock numbers and reproducible artifacts puts it a cut above typical efficiency papers.\n\nThe genuinely new piece is the spatial/temporal head dichotomy for video DiTs, adapted from LLM per-head sparsity work, plus the frame-major layout transformation that makes the strided temporal pattern tensor-core-friendly. The observation that temporal heads are contiguous in the frame-major layout is the most original insight and it is well supported by the attention-map visualizations. The paper is also upfront that the pattern choice is dynamic per prompt and step, which motivates the online profiling.\n\nWhere I am more cautious: the 1% query sampling in Algorithm 1 is the load-bearing assumption. The stress-test concern about rare-but-critical tokens being misclassified is real, but Table 3 gives some comfort: PSNR stays around 31 from 0.1% to 100% profiling on CogVideoX-I2V. That suggests the proxy is not fragile on average. However, the average is on a random VBench subset, and there is no stratification by motion, object interaction, or prompt difficulty. The examples in Figure 6 include a few hard cases, but that is not a systematic test. So the 'preserving generation quality' claim is plausible but slightly stronger than the evidence supports. I would also have liked error bars on the PSNR/SSIM/LPIPS numbers, and a precise list of which VBench prompts were used. These are addressable revision items, not flaws in the core method.\n\nOne minor quibble: Section 3.2 calls the oracle sparse pattern 'lossless' at PSNR 29-30. That is fine relative to the dense baseline, but the label invites over-reading.\n\nWho should engage: anyone working on efficient video diffusion or training-free DiT acceleration. The paper deserves a serious referee. I would accept it for peer review and ask for tighter benchmark reporting, but I would not block on the profiling question without evidence that the proxy actually fails on some prompt class.\n\nNet: send it to review. The central speedup claim holds, the artifacts are there, and the soft spots are mostly around measurement detail rather than conceptual soundness.","headline":"A credible training-free sparse-attention system for video DiTs with real wall-clock speedups; the main unresolved question is how far the 1% profiling proxy generalizes across prompts.","tokens_in":17946,"tokens_out":1658,"would_cite":true,"duration_ms":16074,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Video diffusion transformers can be accelerated more than two-fold without retraining by computing only the attention tokens that matter for each head.","keywords":["video diffusion transformers","sparse attention","spatial-temporal sparsity","attention head classification","training-free acceleration","online profiling","tensor layout transformation","inference speedup"],"falsifier":"Run SVG on prompts with fast-moving objects or visible scene cuts, extract each head's profile-1% decision, and compare it to the oracle decision from full attention on the same step; if any head's choice flips or the generated PSNR falls below the reported ~29-30 on those prompts, the sampling proxy is falsified. A simpler quantitative check is to measure the agreement rate between profile-1% and profile-100% classifications across heads and denoising steps and identify heads where the MSE ranking reverses.","tokens_in":16999,"feed_emoji":"🎬","tokens_out":6707,"duration_ms":57802,"temperature":0.7,"pith_summary":"This paper tries to establish that three-dimensional full attention in video diffusion transformers is dominated by two structured sparse patterns: heads whose output is carried by tokens inside the same frame, and heads whose output is carried by tokens at the same spatial position in other frames. If that is true, then off-the-shelf video DiTs can be accelerated by over 2x with no retraining and with barely any change to the generated video, since only the important tokens need to be computed. The authors back this with an online profiler that samples 1% of query rows per head to decide which pattern to use, plus a layout transformation that turns the temporal pattern into a compact, tensor-core-friendly layout. The result they report is end-to-end speedups of 2.28x on CogVideoX-v1.5 and 2.33x on HunyuanVideo while holding PSNR near 30.","feed_headline":"Sparse attention speeds video diffusion 2.3x with no retraining","feed_subtitle":"A 1% query sample per attention head picks spatial vs temporal sparsity, preserving quality.","key_machinery":"The load-bearing machinery is the spatial/temporal head dichotomy together with the online profiling rule. Spatial Head and Temporal Head are the paper's names for two observed attention-map layouts: block-diagonal (same-frame tokens dominate) and slash-diagonal (same-position-across-frames tokens dominate). The profiler samples 1% of query rows per head, runs full, spatial, and temporal attention on those rows, and picks the sparse pattern with the lowest output MSE versus full attention; this converts the oracle pattern search into a ~3% overhead online decision. The layout transformation then transposes the activations from token-major to frame-major ordering, which makes the temporal head's noncontiguous access pattern contiguous and lets a block-sparse attention kernel reach near-theoretical speedup. Together these pieces carry the argument from observed sparsity to measured end-to-end speedup.","core_discovery":"SVG claims that every attention head in a video diffusion transformer is, at each denoising step, effectively either a Spatial Head (only tokens within the same frame contribute meaningfully, giving a block-diagonal attention map) or a Temporal Head (only tokens at the same spatial location across frames contribute, giving a slash-diagonal map). The method classifies each head cheaply: it computes full, spatial-masked, and temporal-masked attention on a random 1% of query rows, and selects the sparse pattern whose output has lower mean squared error against full attention. It then executes only that pattern, reordering the temporal head's strided token access into a contiguous frame-major layout so GPU tensor cores can use it. On this basis the paper reports preserving generation quality (PSNR above 29) while cutting attention FLOPS to roughly 30%, yielding end-to-end speedups of 2.28x, 2.33x, and 1.51x on CogVideoX-v1.5, HunyuanVideo, and Wan 2.1, with an additional ~1.3x from FP8 quantization on HunyuanVideo.","pith_inferences":["The 1%-sample profiling rule is a bet that a tiny query subset reveals the head's global attention structure; a direct test would be to run SVG on prompts with fast motion or scene cuts, where the same-position temporal correspondence breaks down, and to make the sampling ratio motion-aware.","The spatial/temporal head splitting may transfer to other spatiotemporal transformers, such as video-language encoders, where the same three-dimensional attention layout appears and can be profiled in the same way.","The paper fixes the sparsity ratios cs and ct manually; a natural extension is to tie them to per-scene content such as motion magnitude or resolution, or to cache head types across nearby denoising steps to reduce the 3% profiling overhead further."],"forward_implications":["The reported 2.28x on CogVideoX-v1.5 and 2.33x on HunyuanVideo mean a minutes-long video generation can be cut roughly in half with no retraining and no visible quality loss; with FP8 the HunyuanVideo run drops from 2253s to 968s.","Because the choice of spatial versus temporal pattern is per-head and per-step, the method tracks the dynamic changes in attention during denoising instead of relying on a fixed schedule.","The method composes with orthogonal acceleration techniques: the paper shows FP8 attention adds another ~1.3x throughput at a 0.1 PSNR cost on HunyuanVideo.","The same pipeline on Wan 2.1 reaches 1.51x speedup, indicating the spatial/temporal dichotomy is not specific to one model family."],"supporting_citations":[{"why":"MInference is the LLM sparse-attention baseline that SVG compares against, and whose token-level sparsity fails to capture video's structured patterns.","marker":"(Jiang et al., 2024)"},{"why":"DiTFastAttn provides the spatial-only attention baseline that SVG must beat in both speed and quality.","marker":"(Yuan et al., 2024)"},{"why":"HunyuanVideo is one of the target models; its architecture and benchmark prompts define the main evaluation setting.","marker":"(Kong et al., 2024)"},{"why":"CogVideoX is the other target model; its 3D full attention configuration defines the setup and the per-frame token counts.","marker":"(Yang et al., 2024c)"},{"why":"Provides the optimized full-attention kernel against which the sparse kernels are measured as the baseline.","marker":"(Dao et al., 2022)"},{"why":"Provides the block-sparse attention engine that consumes the transformed layout and delivers the measured speedup.","marker":"(Ye et al., 2025)"},{"why":"VBench supplies the prompt set and quality metrics used to evaluate generation fidelity.","marker":"(Huang et al., 2023)"}],"fun_headline_variants":["Sparse attention classifies heads to speed video diffusion 2.3x","Spatial or temporal? Sparse attention picks per head for 2.3x speedup","Sparse attention speeds video diffusion 2.3x without retraining","Video diffusion 2.3x faster via sparse attention heads"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on 1% of randomly sampled query rows per head being enough to reveal whether that head is spatial or temporal; if for some head, step, or prompt those rows do not show the head's true dominant pattern, the chosen sparse attention will deviate from full attention and reported quality will drop.","fun_headline_variants_meta":{"raw":{"variants":["Sparse attention classifies heads to speed video diffusion 2.3x","Spatial or temporal? Sparse attention picks per head for 2.3x speedup","Sparse attention speeds video diffusion 2.3x without retraining","Video diffusion 2.3x faster via sparse attention heads"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0007,"raw_usage":{"total_tokens":3193,"prompt_tokens":1011,"completion_tokens":2182,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":2098}},"tokens_in":627,"tokens_out":2182,"duration_ms":15134,"temperature":1.0,"reasoning_tokens":2098,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T14:30:01.051570+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SVG on prompts with fast-moving objects or visible scene cuts, extract each head's profile-1% decision, and compare it to the oracle decision from full attention on the same step; if any head's choice flips or the generated PSNR falls below the reported ~29-30 on those prompts, the sampling proxy is falsified. A simpler quantitative check is to measure the agreement rate between profile-1% and profile-100% classifications across heads and denoising steps and identify heads where the MSE ranking reverses.","supporting_citations":[],"review_version":1}