Pith. sign in

REVIEW 4 major objections 5 minor 35 references

Sparse attention speeds video diffusion up to 3.4x, no retraining needed

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

SPADE combines static, semi-static, and dynamic block-sparse attention with a cheap SICS-based blocking heuristic to speed up video diffusion inference by up to 1.80x end-to-end.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Solid systems speedups, but the 'quality preserved' claim only holds when you ignore SPADE-Turbo's fidelity collapse and the policy is hand-tuned on a narrow prompt set. the 4 major comments →

arxiv 2608.03335 v1 pith:CNQ5WVKL submitted 2026-08-04 cs.CV

SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference

Shanghao Liu , Renze Chen , Size Zheng , Yuanqiang Liu , Yun (Eric) Liang , Hailong Yang This is my paper
classification cs.CV
keywords video diffusion transformerssparse attentionblock-sparse attentioninput-adaptive sparsityinference accelerationSICSvDiT-SSRtraining-free acceleration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SPADE is a training-free engine that accelerates video diffusion transformers by replacing dense self-attention with input-adaptive sparse attention. It selects, for each head and timestep, a 3D blocking of tokens whose blocks are most self-similar, then runs block-sparse attention only on those blocks. Across Hunyuan-Video and Wan 2.1/2.2 at 720p, it claims attention speedups of 2.26x to 3.40x and end-to-end speedups of 1.49x to 1.80x while matching or improving VBench quality and staying within 0.01 SSIM of the best sparse baseline.

Core claim

The paper claims that the quadratic cost of self-attention in video diffusion transformers can be made sparse at runtime, per head and per timestep, without any fine-tuning. The key move is to evaluate candidate 3D blockings of the Q/K token grid by a cheap aggregation score, the Sum of Intra-block Cosine Similarities (SICS), which measures how tightly tokens cluster inside each block; the blocking with the highest score is selected, and only the blocks that pass a head-wise, policy-guided Top-K/Top-P and local-window filter are actually attended to. On top of this, a custom flash block-sparse attention kernel with operator fusion and head grouping executes the sparse pattern. The paper repo

What carries the argument

The engine rests on three pieces: (i) vDiT-SSR, a specification that defines 3D blocking candidates (spatial, temporal, mixed) and formalizes dynamic masks through Summarizer/Estimator expressions; (ii) Scheme Generation, which scores each candidate blocking with the online SICS formula (sum of all pairwise cosine similarities inside a block, computed in O(BS·d) via cumulative partial sums) and feeds the selected score plus inference state into a head-wise policy function; and (iii) Head-wise Sparse Attention, an executor with fused layout transform, summarization, and intra-estimation kernels, head grouping, and a flash block-sparse attention kernel using online softmax and Hopper-level fea

Load-bearing premise

The load-bearing premise is that the SICS aggregation score correctly ranks candidate 3D blockings by how well their blocks can be summarized and pruned, and that the two hand-set policy budgets (30:1 for Wan, 80:20 for Hunyuan-Video) transfer across prompts and models without per-prompt tuning.

What would settle it

Run SPADE on a video diffusion transformer not in the evaluated set (e.g., a different backbone or a higher resolution) using the same two fixed policies; if attention speedup falls below the claimed 2.26x floor or VBench drops more than ~0.02 from dense attention, the transferability of the SICS-based policy is refuted. Alternatively, replace SICS with random scores in scheme selection: if quality and speed do not change, then SICS is not the component doing the work.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Video diffusion inference at 720p can be made substantially cheaper without retraining, with attention time cut by more than half and whole-pipeline time cut by roughly a third to a half.
  • The SICS score provides a cheap, online, input-dependent criterion for choosing token blockings, which could replace hand-tuned static windows in other transformer workloads beyond video.
  • A head-wise, timestep-aware policy is sufficient to keep generation quality close to dense attention while pruning 80%+ of attention scores.
  • The separation of policy (scheme generation) from mechanism (sparse executor) means the same engine can be retargeted to new vDiTs by changing only the policy budgets, not the kernels.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If SICS is a robust proxy for how well a block's tokens can be summarized, it may transfer to other sequence-modeling tasks with structured token grids, such as long-context language or multi-view image generation, where the same blocking/summarizing pattern applies.
  • The policy budgets are hand-set per model family; an automated search over those budgets, or a policy that learns from the aggregation score distribution across timesteps, could extend the speedup/quality trade-off frontier beyond the two tested configurations.
  • The reported speedups are hardware-specific (H800); on accelerators with different memory hierarchies the relative gain of block-sparse over dense attention could shift, though the algorithmic sparsity fraction should hold.
  • A testable extension is to replace the SICS ranking with random or reversed rankings in scheme selection; if end-to-end quality barely changes, the aggregation score itself is not the load-bearing component.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. SPADE proposes a training-free sparse-attention engine for video diffusion transformers. It introduces vDiT-SSR, a unified representation for 3D block-sparse masks; SICS, an online per-block cosine-similarity score used to choose a blocking scheme per head; a policy function that maps aggregation scores and inference state to sparsity ratios; and a custom CUDA executor with fused layout transform, index selection, and flash block-sparse attention. On Hunyuan-Video, Wan 2.1/2.2 T2V/I2V at 720p, it reports sparsity of 82–85%, attention speedups of 2.26–3.44×, end-to-end speedups of 1.32–1.49× for SPADE and 1.49–1.80× for SPADE-Turbo, with VBench scores comparable to full attention. The paper claims quality preservation based on VBench plus SSIM/PSNR/LPIPS fidelity metrics.

Significance. If the results hold, SPADE is a useful systems contribution: it demonstrates that input-adaptive block-sparse attention can be made runtime-cheap and kernel-efficient, and the vDiT-SSR abstraction cleanly unifies static, semi-static, and dynamic sparse patterns. Concrete strengths are the real H800 measurements on five open vDiT configurations, the open-source release, and the correct O(BS·d_head) online formulation of SICS in Eq. (5). The main caveat is that the quality-preservation claim is not yet established across the full benchmark: the quality results are single point estimates, the head-wise policy is hand-tuned and validated on a narrow prompt subset, and SPADE-Turbo—the configuration behind the highest end-to-end speedup—shows drastically degraded fidelity to dense attention.

major comments (4)
  1. [§4.2, Table 2] The central quality-preservation claim is not supported by the reported statistics. Table 2 gives one point estimate per metric with no error bars, no number of seeds or prompts, and no per-category breakdown; VBench differences among methods are often 0.01–0.02, within typical run-to-run variation. More seriously, SPADE-Turbo, the configuration needed for the 1.80× end-to-end number, has dramatically lower fidelity than SPADE (e.g., Hunyuan-T2V: SSIM 0.57 vs 0.91, PSNR 15.30 vs 29.04, LPIPS 0.43 vs 0.08) while VBench stays close. Thus the 'preserving quality' claim depends on which metric is used and does not hold for the headline speedup configuration. Please report distributions or error bars and separate SPADE from SPADE-Turbo in the abstract and conclusion.
  2. [§3.2, Eq. (3)-(4); §4.1; §4.5] The scheme-selection heuristic is load-bearing but unvalidated. Eq. (3)-(4) define SICS as an intra-block cosine-sum, and the text asserts that stronger aggregation yields more representative summaries and better dynamic selection, but no proof, bound, or comparison against alternative aggregation criteria (e.g., variance, mean pairwise distance, random candidate) is given. The policy budgets (Func 0 at 30:1, Func 1 at ~80:20, §4.1) are hand-set per model family, and the only direct test of the policy (§4.5) uses Complex_plot prompts. If the aggregation ranking or the budgets do not transfer across VBench-2.0 categories and the three model families, the sparsity and speedup numbers do not demonstrate a general input-adaptive engine. Please add an ablation over aggregation criteria and a policy-sensitivity test on multiple prompt categories and models.
  3. [§4.4, §4.5, Figures 5-6] The supporting evidence for the policy analysis and kernel breakdown is not actually present in the manuscript. Section 4.4 contains unicode escape sequences instead of the attention-time breakdown, and Figure 6's caption is followed by the same corrupted text, so the reader cannot verify the claims that pattern search is 3.48× faster at equal sparsity or that aggregation-guided allocation improves fidelity. These are the only experiments directly validating the two most novel components. Please regenerate the figures/tables and report the actual data.
  4. [Abstract vs. §5 and Table 2] The headline speedup ranges are inconsistent. The abstract states attention 2.26×–3.40× and end-to-end 1.49×–1.80×, while the conclusion states 2.26×–3.44× and 1.32×–1.49× for SPADE, with SPADE-Turbo reaching 1.80×. Table 2 shows SPADE end-to-end speedups of 1.32–1.49×; the 1.49–1.80× range is only achieved by SPADE-Turbo, whose fidelity is substantially worse. The abstract's range conflates the two configurations and overstates the quality-preserving speedup. In addition, the full-attention baseline row in Table 2 appears garbled (e.g., 'Hunyuan-T2V0.77 704 1.00 1.00 0.00%'), making it impossible to check the baseline metrics.
minor comments (5)
  1. [§2.1 and References] The text cites [24] for VAEs, but reference [24] is Wu & He, 'Group Normalization'. The VAE citation appears to be incorrect.
  2. [§2.2, Eq. (2)] The mask M is introduced as 'masking a subset of block pairs', but it is not formally defined. Please specify M as an additive mask taking -inf or 0 on pruned/kept block pairs.
  3. [§3.1] The DSA-SPADE, DSA2-SPADE, and MEAN-SPADE variants are described only in words and a figure; please give the explicit summarizer/estimator expressions so the reader can see how the variants differ.
  4. [§4.1] Please state the number of prompts, seeds, and runs used for each table entry. Without this, small VBench differences in Table 2 are not interpretable.
  5. [Figure 2] The overview figure is dense and the labels are very small; enlarging the figure and defining Summarizer/Estimator terms in the caption would improve readability.

Circularity Check

0 steps flagged

No significant circularity: SPADE's headline claims are measured results, not derived predictions, and no load-bearing argument reduces to its own inputs or to a self-citation.

full rationale

The paper's central claims — sparsity, attention speedup, and end-to-end speedup — are empirical measurements reported in Table 2, obtained by running SPADE on an NVIDIA H800 against dense and sparse baselines. Quality preservation is likewise measured with VBench and fidelity metrics (SSIM/PSNR/LPIPS) relative to full attention, not derived from the method's definitions. SICS (Eqs. 3–5) is a proposed heuristic for ranking blocking candidates by intra-block cosine similarity; its asserted link to selection quality is an unproven assumption, which is a robustness or generalization risk, not a circular step, because SICS is not defined in terms of the reported quality or speed and no result in Table 2 is computed from SICS by construction. The policy budgets (Func 0 and Func 1) are hand-set, and while they are evaluated on the same model families, there is no evidence that the reported quality numbers are statistically forced by fitting those budgets to the same data. The only self-citation (ArkVale [3]) appears in a background taxonomy of dynamic sparse-attention methods and is not load-bearing. No uniqueness theorem is imported, and no known result is merely renamed. Thus the derivation chain is self-contained against external benchmarks for the headline claims; the identified concerns belong to correctness validation, not circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The central claim rests on two heuristic inventions: the SICS aggregation score and the hand-set policy budgets. Both are validated empirically against full attention and baselines, but neither is derived from first principles. The paper introduces no new physical entities. The free parameters are dominated by per-model policy ratios and undisclosed threshold values, which limit claims of transferability.

free parameters (5)
  • Policy budget ratio Func 0 (Wan 2.1/2.2) = 30:1 dynamic-to-static
    Section 4.1: hand-set fixed policy; determines how many blocks are computed per head and is tuned for the two Wan models.
  • Policy budget ratio Func 1 (Hunyuan-Video) = 80:20 dynamic-to-static
    Section 4.1: hand-set per model; no automatic tuning described.
  • Top-K/Top-P thresholds and attention-sink length
    Section 3.1 lists them as utility parameters but no numeric values are given; they directly gate final sparsity and quality.
  • Aggregation-score-to-budget mapping in policy function
    Section 4.5 says budgets are negatively correlated with the aggregation score, but no explicit mapping is provided.
  • Warm-up denoising fraction = 30% T2V, 25% I2V
    Section 4.1, carried from prior work [15,17,25,29]; not derived or varied in this paper.
axioms (6)
  • domain assumption Self-attention is the dominant compute and memory bottleneck in video DiT inference.
    Section 2.2 states attention scales as O(N^2); standard in the literature and used to justify sparsity.
  • domain assumption The vDiT latent space preserves spatio-temporal locality sufficient for blockwise pruning.
    Section 2.1: 'This preserved continuity is key to efficient sparse attention in video generation.'
  • ad hoc to paper Block-level summarizers/estimators (max, min, mean along the head dimension) faithfully represent block importance.
    Section 3.1 defines DSA-SPADE, DSA2-SPADE, and MEAN-SPADE with no derivation; support is empirical only.
  • ad hoc to paper SICS aggregation score ranks candidate blockings by suitability for sparse attention.
    Section 3.2: 'Strong aggregation yields more representative summaries (better dynamic selection)'; asserted without proof and tested only on selected prompts in Section 4.5.
  • domain assumption The first 30% (T2V) or 25% (I2V) of denoising steps must use full attention.
    Section 4.1; adopted from prior work [15,17,25,29], counts as a carried-over modeling choice.
  • standard math FlashAttention-style online softmax is numerically correct, so block-sparse kernels using it produce proper attention outputs.
    Referenced [4,5,19,20]; accepted mechanism from prior literature.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference." pith.science (2026). https://pith.science/paper/CNQ5WVKL

@misc{pith2026260803335,
  author       = {Pith},
  title        = {Pith review of: SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CNQ5WVKL}},
  note         = {Machine review of arXiv:2608.03335}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Video diffusion transformers (vDiTs) generate high quality but pay quadratic self-attention cost, making inference prohibitive at video-token scales. The challenge is input-adaptive sparsity: selecting critical Q/K/V tokens with negligible overhead and executing them for end-to-end gains. We present SPADE, a training-free sparse-attention engine of three parts: (i) vDiT-SSR, a specification defining 3D blocking candidates and formalizing dynamic masks via Summarizer/Estimator expressions; (ii) runtime scheme generation using SICS and a head-wise policy; and (iii) an executor with low-overhead index search, flash block-sparse attention, and kernel grouping. Across Hunyuan-Video and Wan 2.1/2.2 for text-to-video and image-to-video generation, SPADE raises sparsity and speed while preserving quality, accelerating attention by 2.26x-3.40x and end-to-end inference by 1.49x-1.80x. Our code is open-sourced at https://github.com/6somehow/DAC-SPADE.

Figures

Figures reproduced from arXiv: 2608.03335 by Hailong Yang, Renze Chen, Shanghao Liu, Size Zheng, Yuanqiang Liu, Yun (Eric) Liang.

Figure 1
Figure 1. Figure 1: Illustration of the variational autoencoder (VAE) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of SPADE (Sparse Video DiT Engine). [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of blocking schemes (order: T,H,W): (a) spatial (1,8,8); (b) temporal (16,1,4); (c) mixed (2,4,8). Each color [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of different summarizer and estimator expressions. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Attention time breakdown of sparse attention meth [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Sensitivity test on Sparse-VideoGen2 and SPADE [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 9 canonical work pages

  1. [1]

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. 2024. PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling. arXiv:2406.02069 [cs] http://arxiv.org/abs/2406.02069

    Pith/arXiv arXiv 2024

  2. [2]

    Pengtao Chen, Xianfang Zeng, Maosen Zhao, Peng Ye, Mingzhu Shen, Wei Cheng, Gang Yu, and Tao Chen. 2025. Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers. doi:10.48550/arXiv.2506. 03065 arXiv:2506.03065 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2506 2025

  3. [3]

    Renze Chen, Zhuofeng Wang, Beiquan Cao, Tong Wu, Size Zheng, Xiuhong Li, Xuechao Wei, Shengen Yan, Meng Li, and Yun Liang. 2024. Arkvale: Efficient generative llm inference with recallable key-value eviction.Advances in Neural Information Processing Systems37 (2024), 113134–113155

    2024

  4. [4]

    Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. InInternational Conference on Learning Representations (ICLR)

    2024

  5. [5]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv:2205.14135 [cs] http://arxiv.org/abs/2205.14135

    Pith/arXiv arXiv 2022

  6. [6]

    Yichuan Deng, Zhao Song, and Chiwun Yang. 2024. Attention is Naturally Sparse with Gaussian Distributed Input. arXiv:2404.02690 [cs] http://arxiv.org/abs/2404. 02690

    Pith/arXiv arXiv 2024

  7. [7]

    Hangliang Ding, Dacheng Li, Runlong Su, Peiyuan Zhang, Zhijie Deng, Ion Stoica, and Hao Zhang. 2025. Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile. doi:10.48550/arXiv.2502.06155 arXiv:2502.06155 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.06155 2025

  8. [8]

    Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Peiyuan Zhou, Jiaxing Qi, Junjie Lai, Hayden Kwok-Hay So, Ting Cao, Fan Yang, and Mao Yang. 2025. Seer- Attention: Learning Intrinsic Sparse Attention in Your LLMs. arXiv:2410.13276 [cs] doi:10.48550/arXiv.2410.13276

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.13276 2025

  9. [9]

    Google. [n. d.]. https://aistudio.google.com/models/veo-3

    work page

  10. [10]

    Junxian Guo, Haotian Tang, Shang Yang, Zhekai Zhang, Zhijian Liu, and Song Han. 2024. Block Sparse Attention. https://github.com/mit-han-lab/Block-Sparse- Attention

    work page 2024

  11. [11]

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuan- han Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. 2024. Vbench: Comprehensive benchmark suite for video generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 21807– 21818

    2024

  12. [12]

    Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention. arXiv:2407.02490 [cs] http://arxiv.org/abs/ 2407.02490

    Pith/arXiv arXiv 2024

  13. [13]

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, Junkun Yuan, Yanxin Long, Aladdin Wang, Andong Wang, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Hongmei Wang, Jacob Song, Jiawang Bai, Jianbing Wu, Jinbao Xue, Joey Wang, Kai Wang, Mengyang Liu, Pengyu Li, Shuai Li, ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.03603 2025

  14. [14]

    Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. 2025. FlexPre- fill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference. arXiv:2502.20766 [cs] doi:10.48550/arXiv.2502.20766

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.20766 2025

  15. [15]

    Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Kai Li, and Song Han. 2024. Distrifusion: Distributed parallel inference for high-resolution diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7183–7193

    work page 2024

  16. [16]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. SnapKV: LLM Knows What You are Looking for Before Generation. arXiv:2404.14469 [cs] http://arxiv.org/abs/2404.14469

    Pith/arXiv arXiv 2024

  17. [17]

    Zhengyao Lv, Chenyang Si, Junhao Song, Zhenyu Yang, Yu Qiao, Ziwei Liu, and Kwan-Yee K Wong. 2024. Fastercache: Training-free video diffusion model acceleration with high quality.arXiv preprint arXiv:2410.19355(2024)

    Pith/arXiv arXiv 2024

  18. [18]

    OpenAI. [n. d.]. https://openai.com/index/sora-2/

    work page

  19. [19]

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. 2025. FlashAttention-3: fast and accurate attention with asynchrony and low- precision. InProceedings of the 38th International Conference on Neural Information Processing Systems(Vancouver, BC, Canada)(NIPS ’24). Curran Associates Inc., Red Hook, NY, USA, Article 2193, 28 pages

    work page 2025

  20. [20]

    Benjamin F Spector, Simran Arora, Aaryan Singhal, Daniel Y Fu, and Christopher Ré. 2024. Thunderkittens: Simple, fast, and adorable ai kernels.arXiv preprint arXiv:2410.20399(2024)

    Pith/arXiv arXiv 2024

  21. [21]

    Xin Tan, Yuetao Chen, Yimin Jiang, Xing Chen, Kun Yan, Nan Duan, Yibo Zhu, Daxin Jiang, and Hong Xu. 2025. DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training. doi:10.48550/arXiv.2502.07590 arXiv:2502.07590 [cs]

    work page doi:10.48550/arxiv.2502.07590 2025

  22. [22]

    Guoxia Wang, Jinle Zeng, Xiyuan Xiao, Siming Wu, Jiabin Yang, Lujing Zheng, Zeyu Chen, Jiang Bian, Dianhai Yu, and Haifeng Wang. 2024. FlashMask: Efficient and Rich Mask Extension of FlashAttention.arXiv preprint arXiv:2410.01359 (2024)

    Pith/arXiv arXiv 2024

  23. [23]

    WanTeam, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Ti...

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.20314 2025

  24. [24]

    Yuxin Wu and Kaiming He. 2018. Group normalization. InProceedings of the European conference on computer vision (ECCV). 3–19

    2018

  25. [25]

    Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, Jianfei Chen, Ion Stoica, Kurt Keutzer, and Song Han. 2025. Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity. arXiv:2502.01776 [cs] doi:10.48550/arXiv.2502. 01776

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.01776 2025

  26. [26]

    Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. 2024. DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads. arXiv:2410.10819 [cs] http: //arxiv.org/abs/2410.10819

    Pith/arXiv arXiv 2024

  27. [27]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient Streaming Language Models with Attention Sinks. arXiv:2309.17453 [cs] http://arxiv.org/abs/2309.17453

    Pith/arXiv arXiv 2024

  28. [28]

    Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. 2025. XAttention: Block Sparse Attention with Antidiagonal Scoring. doi:10.48550/ arXiv.2503.16428 arXiv:2503.16428 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.16428 2025

  29. [29]

    Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Jintao Zhang, Han Cai, Yujun Lin, Xiuyu Li, Chenfeng Xu, Kelly Peng, Jianfei Chen, Song Han, Kurt Keutzer, and Ion Stoica. 2025. Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation. doi:10.48550/arXiv.2505. 18875 arXiv:2505.18875 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2505 2025

  30. [30]

    Zhihang Yuan, Hanling Zhang, Lu Pu, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. 2024. Ditfastattn: Attention compression for diffusion transformer models.Advances in Neural Information Processing Systems37 (2024), 1196–1219

    work page 2024

  31. [31]

    Hanling Zhang, Rundong Su, Zhihang Yuan, Pengtao Chen, Mingzhu Shen Yibo Fan, Shengen Yan, Guohao Dai, and Yu Wang. 2025. DiTFastAttnV2: Head- wise Attention Compression for Multi-Modality Diffusion Transformers.arXiv preprint arXiv:2503.22796(2025)

    Pith/arXiv arXiv 2025

  32. [32]

    Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. 2025. SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference. arXiv:2502.18137 [cs] doi:10.48550/arXiv.2502.18137

    work page doi:10.48550/arxiv.2502.18137 2025

  33. [33]

    Peiyuan Zhang, Yongqi Chen, Haofeng Huang, Will Lin, Zhengzhong Liu, Ion Stoica, Eric Xing, and Hao Zhang. 2025. VSA: Faster Video Diffusion with Trainable Sparse Attention. doi:10.48550/arXiv.2505.13389 arXiv:2505.13389 [cs]

    work page doi:10.48550/arxiv.2505.13389 2025

  34. [34]

    Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhenghong Liu, and Hao Zhang. 2025. Fast Video Generation with Sliding Tile Attention. arXiv:2502.04507 [cs] doi:10.48550/arXiv.2502.04507

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.04507 2025

  35. [35]

    Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Lulu Gu, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, et al. 2025. Vbench-2.0: Advanc- ing video generation benchmark suite for intrinsic faithfulness.arXiv preprint arXiv:2503.21755(2025)

    Pith/arXiv arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.