Pith. sign in

REVIEW 2 major objections 5 minor 52 references

Sol-Attn fuses dynamic routing, sparse computation, and approximation correction into a single online-softmax pass, delivering 1.9–3.0× speedups for video generation without retraining.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:14 UTC pith:PXZGXECN

load-bearing objection A real engineering advance, modestly oversold in the abstract; the pooled-mean approximation is the main soft spot but the empirical case is strong enough for peer review. the 2 major comments →

arxiv 2607.24027 v1 pith:PXZGXECN submitted 2026-07-27 cs.CV

Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification

Haopeng Li , Yitong Li , Junsong Chen , Tian Ye , Haozhe Liu , Jincheng Yu , Duomin Wang , Ruihua Zhang
show 3 more authors
Zeke Xie Enze Xie Song Han
This is my paper
classification cs.CV
keywords sparse attentiondiffusion transformervideo generationonline softmaxblock sparsitytraining-free accelerationapproximate attentioninference speedup
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Sol-Attn claims that block-sparse attention for diffusion video transformers can be made both faster and more accurate by removing the separate routing stage. It selects key-value blocks on the fly using a query-dependent threshold derived from the mean and standard deviation of block proxy scores, and reuses the proxy scores of unselected blocks as zeroth-order Taylor approximations of their contribution. All of this happens inside a single online-softmax kernel, so no proxy score map or routing indices are ever written to memory. On H100 GPUs, the paper reports end-to-end speedups of 1.9–3.0× over dense FlashAttention-3 baselines while keeping VBench quality and dense-reference similarity close to or better than prior sparse methods. If correct, Sol-Attn is a drop-in, training-free acceleration for bidirectional diffusion transformers in video and image generation.

Core claim

The paper's central claim is that block selection, exact sparse computation, and approximate correction of skipped blocks can coexist in one streaming online-softmax pass. Given that block-proxy logits in pretrained video models are near-Gaussian, a standardized cutoff β maps to a controlled overall density; each query row converts this cutoff into a raw threshold τ_i = μ_i + βσ_i using row-wise mean and standard deviation computed cheaply from pooled-key moments. As the outer loop streams pooled-key chunks, the token-to-block score tile simultaneously routes blocks above threshold to an exact inner loop over the original key-value tiles and contributes below-threshold columns to the softmax

What carries the argument

The carrying mechanism is the fused nested-loop online softmax with proxy-score reuse: a query block scans pooled-key chunks, the column means of the resulting token-to-block score tile are compared on-chip against τ_i, selected blocks are dispatched to a sparse exact attention loop over original key-value tiles, and unselected columns stay in registers as zeroth-order Taylor approximations exp(Q_i \bar{K}_j^T). The same score tile serves routing, approximation, and exact computation, with one online-softmax state shared by both branches; the threshold τ_i is computed from first and second moments of the pooled keys, avoiding a full proxy map.

Load-bearing premise

The load-bearing premise is that unselected 64×64 key blocks are faithfully represented by their pooled mean for both routing and correction; if a block contains a few highly attended tokens but a low mean proxy score, it is dropped and its true contribution is under-approximated.

What would settle it

Measure the exact attention output for query rows where a key block's mean proxy score lies below threshold but the block contains one or two tokens with very high attention scores, then compute the output error against dense attention; Appendix B shows the error grows with the centered score spread Σ δ_u^2. A systematic pattern of such high-variance blocks in a trained video model would degrade the quality–sparsity trade-off beyond what the paper's aggregate metrics reveal.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • At matched sparsity, Sol-Attn reports end-to-end speedups of 2.02×, 2.12×, 2.4× (stage-2), 3.04×, and 2.34× on Wan2.1, HunyuanVideo, LTX, SANA-WM, and Bernini respectively, with VBench and similarity metrics at or above competing sparse methods.
  • Routing overhead drops by 11.5× and 32.7× compared to top-k and top-p routing, and attention-processor memory stays near dense attention instead of the ~8× overhead of routing-based methods.
  • Approximate correction consistently reduces output error: at 90% sparsity it lowers relative ℓ2 error and preserves higher cosine similarity relative to exact-only sparse attention under identical block selections.
  • Because the method is training-free and kernel-level, it composes with complementary techniques like diffusion-step caching and kernel fusion, reaching end-to-end speedups of 3.48× and 5.08× when combined in the Sol-Engine system.
  • The same unified pass works across text-to-video, video-to-video editing, refinement, and 2K text-to-image generation, suggesting the mechanism is not model-specific.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The Gaussian-calibration assumption implies a testable extension: per-layer or per-head β schedules could trade quality for speed more finely; the paper shares one β per model, but its own density plots show per-row variance across layers.
  • Upgrading the approximation from zeroth-order to a first-order correction using within-block key–value covariance could reduce error on high-variance blocks at extra FLOP cost; the paper's Appendix B bound shows numerator error depends on the alignment between score deviations and value magnitudes.
  • If the near-Gaussian proxy-score structure holds for bidirectional attention more broadly, the same on-the-fly thresholding could transfer to other bidirectional sequence tasks, but the paper only evaluates diffusion-based visual generation models.
  • The authors note the kernel supports only forward inference; a backward pass would need to record which blocks were selected, making training or fine-tuning with Sol-Attn a distinctly different, open problem.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes Sol-Attn, a training-free block-sparse attention method for bidirectional diffusion video/image transformers. It replaces top-k/top-p routing with a query-dependent threshold τ_i = μ_i + βσ_i applied on-the-fly to pooled-key block proxy scores during online softmax, avoiding materialization of the proxy map. Unselected blocks are not discarded; their block-wise contribution is approximated by a zeroth-order Taylor expansion using the pooled key (Eqs. 8–10). The fused kernel is evaluated on H100 (and RTX 5090, B200) on Wan, Hunyuan, LTX, Bernini, SANA-WM, and Ideogram, reporting end-to-end speedups vs FlashAttention-3/4 of 1.9–3.0× at ~85–90% sparsity, with VBench/PSNR/SSIM/LPIPS close to dense attention. Appendix B derives the threshold variance and an approximation-error bound.

Significance. If the quality claims hold, Sol-Attn is a practically useful drop-in acceleration with a clean design: it introduces no fitted parameters (β is an explicit sparsity knob), the routing overhead is largely fused into the online-softmax pipeline, and the approximate correction is principled. The broad benchmark coverage, kernel-level latency breakdowns, and the honest error bound in Eq. (16) are strengths. The main risk is that the entire accuracy mechanism rests on the pooled-mean block model, whose intra-block variance is never directly measured.

major comments (2)
  1. [§3.2, Eq. (8), Appendix B Eq. (16)] The central accuracy mechanism is the pooled-mean block model: Eq. (3)/(4) routes on \bar{Q}_i \bar{K}_j^T and Eq. (8) approximates exp(Q_i K_j^T) by exp(Q_i \bar{K}_j^T) for unselected blocks. Appendix B, Eq. (16), bounds the denominator error by (1/2)exp(a+η)Σδ_u^2 and the numerator error by exp(a+η)Σ|δ_u|||v_{j,u}||. These bounds grow with intra-block centered score spread, which the paper never measures on dropped blocks. The Gaussian calibration in Fig. 3 is over the aggregate standardized proxy distribution, not per-block key heterogeneity; Fig. 9 reports average full-output ℓ2/cosine, which can mask localized failures where a dropped block contained a few highly attended tokens. Please report the distribution of Σδ_u^2 over dropped blocks for the evaluated models, or provide an ablation with synthetic high-variance key blocks, and discuss worst-case output error. Without this, the
  2. [Abstract, Table 1, Table 3] The headline speedups are not consistently reproduced in the tables. The Abstract states '2.1× and 2.3× end-to-end speedups for video generation and editing, respectively'; the Introduction states '2.1×–3.0× across video tasks.' However, Table 1 reports end-to-end speedups of 2.02× (Wan), 2.12× (Hunyuan), and 1.9× (LTX), and Table 3 reports 2.34× for Bernini. The 2.1×/2.3× pairing appears to select Hunyuan and Bernini; the LTX result is below the abstract's '2.1×' and the '–3.0×' range is only reached by the SANA-WM refiner (Table 2, 3.04×). Please clarify which tasks/settings the abstract and introduction refer to, or revise the claims to match the tables.
minor comments (5)
  1. [§3.2, Algorithm 1] The phrase 'single online-softmax pass' is slightly overstated: τ_i is computed from Eq. (5) as a separate pre-pass over pooled-key statistics (Algorithm 1 receives tau[i] as input). Consider describing it as 'nearly single-pass' or 'with a lightweight pre-pass.'
  2. [Tables 1–3] Sparsity matching across baselines is approximate: in Table 1, Wan sparsities are 85.60%, 83.66%, 85.00%, 85.12%; LTX sparsities are 90.17%, 89.35%, 90.00%, 89.83%. Differences up to ~2 percentage points could affect speedup/quality comparisons. State a tolerance or re-run at identical sparsity.
  3. [Figure 2] The caption lists 'Ours' sparsity values 75.0% and 67.2%, which do not match the values reported in Table 1 (e.g., 85.12% for Wan). Please make the example consistent with the experiments.
  4. [Algorithm 1] The arrays KC and VC are not defined before use in the pseudocode. Define them (e.g., pooled keys and summed values) in the algorithm or caption.
  5. [Abstract and Figure 3] Minor typographical issues: 'training-freeSol-Attn' appears without a space in the abstract; the x-axis of Fig. 3 is labeled 'cutoffβ(right)' awkwardly; and the phrase 'on-the-fly attention' should be 'on-the-fly attention sparsification' or similar.

Circularity Check

0 steps flagged

No significant circularity: Sol-Attn's routing, approximation, and fused online-softmax derivation are self-contained and externally benchmarked; self-citations are not load-bearing.

full rationale

Sol-Attn's derivation chain is self-contained rather than circular. The block-proxy scores in Eq. (3), the threshold in Eq. (4), and the moment-based threshold computation in Eq. (5) are all defined directly from the pooled-key statistics; no quantity is fitted to the quality metric it later claims to predict. Eq. (8) is an explicit Taylor expansion whose zeroth-order term is the pooled-mean approximation, and Eqs. (9)-(10) combine exact and approximate terms in the same online-softmax state. Eq. (11) is an exact algebraic identity (the column mean of the token-to-block score tile equals the block-proxy score), not a fitted prediction. The hyperparameter beta is an explicit user-controlled sparsity knob, calibrated by the empirical near-Gaussian observation in Fig. 3 rather than a hidden parameter fitted to accuracy. The main accuracy claims are validated against dense FlashAttention baselines and external benchmarks (VBench, Bernini-Bench, Qwen-Image-Bench, PSNR/SSIM/LPIPS), so the central claim does not reduce to its inputs. The self-citations to PISA and Sol-Engine are disclosed prior work used for comparison and integration context, not as load-bearing proof of Sol-Attn's own approximation. The paper's own Appendix B honestly states the approximation error bound (Eq. 16), which depends on the centered score spread within key blocks; whether that spread is small in practice is an empirical correctness risk, not a circularity in the derivation.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim rests on three domain assumptions (Gaussian proxy-score distribution, smoothness inside key blocks, mean-proxy importance) and on hand-set knobs (β, block size, warm-up fraction, chunk size). No new physical or latent entities are introduced.

free parameters (4)
  • β (standardized cutoff) = not reported; calibrated per model to reach target sparsity (≈85–90%)
    Threshold τ_i = μ_i + βσ_i; β is the sparsity control and is tuned per model to match baselines' sparsity in the experiments (Table 1). It is not derived from first principles.
  • block size B = 64×64 physical blocks
    Block granularity for exact sparse attention; chosen for hardware alignment and affects proxy quality and achievable speedup.
  • dense warm-up fraction = 20% of denoising steps + first layer (most models)
    Dense attention during the first 20% of steps and first layer; hand-chosen following prior work and directly affects end-to-end speedup and quality.
  • chunk size C for routing/approximation = not reported
    Affects kernel efficiency but not mathematical output per the paper; still a free implementation parameter for the speedup claim.
axioms (4)
  • domain assumption Standardized block-proxy logits are approximately Gaussian within each model
    Used to map β to selected density via 1−Φ(β) and to claim controllable dynamic budgets (Sec. 3.1, Fig. 3). This is an empirical observation, not a theorem.
  • domain assumption Unselected key blocks can be approximated by their pooled mean key (zeroth-order Taylor) with bounded error
    Eq. (8) and Appendix B; the quality of Sol-Attn's correction depends on attention logits being smooth within each block and on the bounds in Eq. (16).
  • domain assumption Block-level mean proxy score is a valid importance signal for selecting critical key blocks
    Eq. (3) and Algorithm 1; if important tokens live in blocks with low mean proxy, those blocks are dropped.
  • standard math Online softmax with exact and approximate updates produces a normalized output
    FlashAttention online softmax (Sec. 2) and the fact that selected blocks are included exactly and unselected blocks approximately, with no double counting.

pith-pipeline@v1.3.0-alltime-deepseek · 17822 in / 18038 out tokens · 168924 ms · 2026-07-31T23:14:19.037775+00:00 · methodology

0 comments
read the original abstract

Diffusion transformers are essential for high-fidelity video generation, but long token sequences make attention a dominant inference bottleneck. Training-free dynamic sparse attention alleviates this bottleneck by computing only selected key-value blocks, yet existing methods struggle to sparsify attention both efficiently and accurately for two reasons: (1) Rigid, unpredictable, and costly routing: selecting a fixed fraction of top-ranked blocks by proxy score imposes fixed budgets, whereas retaining blocks to reach a target cumulative proxy probability mass yields dynamic but potentially imbalanced budgets; both incur non-negligible overhead from computing and materializing proxy scores. (2) Lossy keep-or-drop sparsification: unselected blocks are discarded entirely, degrading accuracy under aggressive sparsity. These limitations motivate cheaper dynamic-budget routing while limiting accuracy degradation. In this paper, we introduce training-free Sol-Attn (Sparsifying online attention), which unifies dynamic routing, sparse computation, and approximation correction in a single online-softmax pass, achieving a better accuracy-efficiency trade-off in sparse attention. The core of Sol-Attn is on-the-fly block thresholding with proxy-score reuse, which selects critical blocks by comparing block proxy scores against a threshold during online softmax. This design enables dynamic yet controllable block budgets without materializing the proxy map, while directly reusing the proxy scores of unselected blocks to approximate their contribution. Experiments across image and video generation tasks show that Sol-Attn advances the quality-efficiency frontier of training-free sparse attention, delivering 2.1 times and 2.3 times end-to-end speedups for video generation and editing, respectively, while preserving visual quality.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 13 linked inside Pith

  1. [1]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023

    2023

  2. [2]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    HunyuanVideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. HunyuanVideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    LTX-Video: Realtime video latent diffusion.arXiv preprint arXiv:2501.00103, 2024

    Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, Poriya Panet, Sapir Weissbuch, Victor Kulikov, Yaki Bitterman, Zeev Melumian, and Ofir Bibi. LTX-Video: Realtime video latent diffusion.arXiv preprint arXiv:2501.00103, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    LTX-2.3 Model Card

    Lightricks. LTX-2.3 Model Card. https://huggingface.co/Lightricks/LTX-2.3, 2026. Model checkpoint family including ltx-2.3-22b-dev and distilled variants

    2026

  6. [6]

    MaineCoon: Pursuing a real-time audio-visual social world model.arXiv preprint arXiv:2606.17800, 2026

    Lichen Bai, Tianhao Zhang, Shitong Shao, Dingwei Tan, Qiyu Zhong, Zhengpeng Xie, Haopeng Li, Qinghao Huang, Dandan Shen, Tengjiao Ji, et al. MaineCoon: Pursuing a real-time audio-visual social world model.arXiv preprint arXiv:2606.17800, 2026

    Pith/arXiv arXiv 2026

  7. [7]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  8. [8]

    Sparse VideoGen2: Accelerate video generation with sparse attention via semantic-aware permutation

    Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Jintao Zhang, Han Cai, Yujun Lin, Xiuyu Li, Chenfeng Xu, Kelly Peng, Jianfei Chen, Song Han, Kurt Keutzer, and Ion Stoica. Sparse VideoGen2: Accelerate video generation with sparse attention via semantic-aware permutation. InAdvances in Neural Information Processing Systems (NeurIPS), 2025

    2025

  9. [9]

    SpargeAttention: Accurate and training-free sparse attention accelerating any model inference

    Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. SpargeAttention: Accurate and training-free sparse attention accelerating any model inference. InProceedings of the 42nd International Conference on Machine Learning (ICML), 2025

    2025

  10. [10]

    XAttention: Block sparse attention with antidiagonal scoring

    Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. XAttention: Block sparse attention with antidiagonal scoring. InProceedings of the 42nd International Conference on Machine Learning (ICML), 2025

    2025

  11. [11]

    PISA: Piecewise sparse attention is wiser for efficient diffusion transformers.arXiv preprint arXiv:2602.01077, 2026

    Haopeng Li, Shitong Shao, Wenliang Zhong, Zikai Zhou, Lichen Bai, Hui Xiong, and Zeke Xie. PISA: Piecewise sparse attention is wiser for efficient diffusion transformers.arXiv preprint arXiv:2602.01077, 2026

    arXiv 2026

  12. [12]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-Awareness. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  13. [13]

    LIVEditor-14B: Lightning unified video editing via in-context sparse attention

    Shitong Shao, Zikai Zhou, Haopeng Li, Yingwei Song, Wenliang Zhong, Lichen Bai, and Zeke Xie. LIVEditor-14B: Lightning unified video editing via in-context sparse attention. InProceedings of the 43rd International Conference on Machine Learning (ICML), 2026

    2026

  14. [14]

    Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  15. [15]

    Optimizing mixture of block attention.arXiv preprint arXiv:2511.11571, 2025

    Guangxuan Xiao, Junxian Guo, Kasra Mazaheri, and Song Han. Optimizing mixture of block attention.arXiv preprint arXiv:2511.11571, 2025

    arXiv 2025

  16. [16]

    Bernini: Latent semantic planning for video diffusion.arXiv preprint arXiv:2605.22344, 2026

    Chenchen Liu, Junyi Chen, Lei Li, Lu Chi, Mingzhen Sun, Zhuoying Li, Yi Fu, Ruoyu Guo, Yiheng Wu, Ge Bai, and Zehuan Yuan. Bernini: Latent semantic planning for video diffusion.arXiv preprint arXiv:2605.22344, 2026

    Pith/arXiv arXiv 2026

  17. [17]

    SANA-WM: Efficient minute-scale world modeling with hybrid linear diffusion transformer.arXiv preprint arXiv:2605.15178, 2026

    Haoyi Zhu, Haozhe Liu, Yuyang Zhao, Tian Ye, Junsong Chen, Jincheng Yu, Tong He, Song Han, and Enze Xie. SANA-WM: Efficient minute-scale world modeling with hybrid linear diffusion transformer.arXiv preprint arXiv:2605.15178, 2026

    Pith/arXiv arXiv 2026

  18. [18]

    Ideogram 4.https://ideogram.ai/blog/ideogram-4.0/, 2026

    Ideogram AI. Ideogram 4.https://ideogram.ai/blog/ideogram-4.0/, 2026

    2026

  19. [19]

    VBench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

    2024

  20. [20]

    Qwen-Image-Bench: From generation to creation in text-to-image evaluation.arXiv preprint arXiv:2605.28091, 2026

    Niantong Li, Guangzheng Hu, Weixu Qiao, Ying Ba, Qichen Hong, et al. Qwen-Image-Bench: From generation to creation in text-to-image evaluation.arXiv preprint arXiv:2605.28091, 2026

    Pith/arXiv arXiv 2026

  21. [21]

    FlashAttention-3: Fast and accurate attention with asynchrony and low-precision

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  22. [22]

    FlashAttention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling.arXiv preprint arXiv:2603.05451, 2026

    Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, Vijay Thakkar, and Tri Dao. FlashAttention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling.arXiv preprint arXiv:2603.05451, 2026

    arXiv 2026

  23. [23]

    Less is enough: Training-free video diffusion acceleration via runtime-adaptive caching.arXiv preprint arXiv:2507.02860, 2025

    Xin Zhou, Dingkang Liang, Kaijin Chen, Tianrui Feng, Xiwu Chen, Hongkai Lin, Yikang Ding, Feiyang Tan, Hengshuang Zhao, and Xiang Bai. Less is enough: Training-free video diffusion acceleration via runtime-adaptive caching.arXiv preprint arXiv:2507.02860, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    Timestep embedding tells: It’s time to cache for video diffusion model

    Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, and Fang Wan. Timestep embedding tells: It’s time to cache for video diffusion model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  25. [25]

    From reusing to forecasting: Accelerating diffusion models with TaylorSeers

    Jiacheng Liu, Chang Zou, Yuanhuiyi Lyu, Junjie Chen, and Linfeng Zhang. From reusing to forecasting: Accelerating diffusion models with TaylorSeers. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025

    2025

  26. [26]

    Sol video inference engine: Agent-native full-stack acceleration framework for efficient video generation.arXiv preprint arXiv:2606.23743, 2026

    Yitong Li, Junsong Chen, Haopeng Li, Haozhe Liu, Jincheng Yu, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Sol video inference engine: Agent-native full-stack acceleration framework for efficient video generation.arXiv preprint arXiv:2606.23743, 2026

    Pith/arXiv arXiv 2026

  27. [27]

    Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019

    Pith/arXiv arXiv 1904

  28. [28]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150, 2020

    Pith/arXiv arXiv 2004

  29. [29]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. InAdvances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  30. [30]

    Reformer: The efficient transformer

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. InInternational Conference on Learning Representations (ICLR), 2020

    2020

  31. [31]

    Efficient content-based sparse attention with routing transformers.Transactions of the Association for Computational Linguistics, 9:53–68, 2021

    Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers.Transactions of the Association for Computational Linguistics, 9:53–68, 2021

    2021

  32. [32]

    SeerAttention: Self-distilled attention gating for efficient long-context prefilling

    Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Peiyuan Zhou, Jiaxing Qi, Junjie Lai, Hayden So, Ting Cao, Fan Yang, and Mao Yang. SeerAttention: Self-distilled attention gating for efficient long-context prefilling. InAdvances in Neural Information Processing Systems (NeurIPS), 2025

    2025

  33. [33]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attention. InProceedings of the 63rd Annual Meeting of the Association for Computational...

    2025

  34. [34]

    MoBA: Mixture of block attention for long-context LLMs

    Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Yutao Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu. MoBA: Mixture of block attention for long-contex...

    2025

  35. [35]

    Sparse video-gen: Accelerating video diffusion transformers with spatial-temporal sparsity

    Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, Jianfei Chen, Ion Stoica, Kurt Keutzer, and Song Han. Sparse video-gen: Accelerating video diffusion transformers with spatial-temporal sparsity. InProceedings of the 42nd International Conference on Machine Learning (ICML), 2025

    2025

  36. [36]

    Radial attention:𝒪(𝑛log𝑛)sparse attention with energy decay for long video generation

    Xingyang Li, Muyang Li, Tianle Cai, Haocheng Xi, Shuo Yang, Yujun Lin, Lvmin Zhang, Songlin Yang, Jinbo Hu, Kelly Peng, Maneesh Agrawala, Ion Stoica, Kurt Keutzer, and Song Han. Radial attention:𝒪(𝑛log𝑛)sparse attention with energy decay for long video generation. InAdvances in Neural Information Processing Systems (NeurIPS), 2025

    2025

  37. [37]

    Fast video generation with sliding tile attention

    Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhengzhong Liu, and Hao Zhang. Fast video generation with sliding tile attention. InProceedings of the 42nd International Conference on Machine Learning (ICML), 2025. 15 Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification

    2025

  38. [38]

    Sparse-vDiT: Unleashing the power of sparse attention to accelerate video diffusion transformers

    Pengtao Chen, Xianfang Zeng, Maosen Zhao, Mingzhu Shen, Wei Cheng, Gang Yu, and Tao Chen. Sparse-vDiT: Unleashing the power of sparse attention to accelerate video diffusion transformers. InProceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2026

    2026

  39. [39]

    Xing, and Hao Zhang

    Peiyuan Zhang, Yongqi Chen, Haofeng Huang, Will Lin, Zhengzhong Liu, Ion Stoica, Eric P. Xing, and Hao Zhang. Faster video diffusion with trainable sparse attention. InAdvances in Neural Information Processing Systems (NeurIPS), 2025

    2025

  40. [40]

    DFSAttn: Dynamic fine-grained sparse attention for efficient video generation

    Jie Hu, Zixiang Gao, Yutong He, and Kun Yuan. DFSAttn: Dynamic fine-grained sparse attention for efficient video generation. InProceedings of the 43rd International Conference on Machine Learning (ICML), 2026

    2026

  41. [41]

    FPSAttention: Training-aware FP8 and sparsity co-design for fast video diffusion

    Akide Liu, Zeyu Zhang, Zhexin Li, Xuehai Bai, Yuanjie Xing, Yizeng Han, Jiasheng Tang, Jichao Wu, Mingyang Yang, Weihua Chen, Jiahao He, Yuanyu He, Fan Wang, Reza Haffari, and Bohan Zhuang. FPSAttention: Training-aware FP8 and sparsity co-design for fast video diffusion. InAdvances in Neural Information Processing Systems (NeurIPS), 2025

    2025

  42. [42]

    SpargeAttention2: Trainable sparse attention via hybrid top-𝑘+top-𝑝 masking and distillation fine-tuning.arXiv preprint arXiv:2602.13515, 2026

    Jintao Zhang, Kai Jiang, Chendong Xiang, Weiqi Feng, Yuezhou Hu, Haocheng Xi, Jianfei Chen, and Jun Zhu. SpargeAttention2: Trainable sparse attention via hybrid top-𝑘+top-𝑝 masking and distillation fine-tuning.arXiv preprint arXiv:2602.13515, 2026

    arXiv 2026

  43. [43]

    Twilight: Adaptive attention sparsity with hierarchical top-𝑝 pruning

    Chaofan Lin, Jiaming Tang, Shuo Yang, Hanshuo Wang, Tian Tang, Boyu Tian, Ion Stoica, Song Han, and Mingyu Gao. Twilight: Adaptive attention sparsity with hierarchical top-𝑝 pruning. InAdvances in Neural Information Processing Systems (NeurIPS), 2025

    2025

  44. [44]

    VecAttention: Vector- wise sparse attention for accelerating long context inference

    Anmin Liu, Ruixuan Yang, Huiqiang Jiang, Bin Lin, Minmin Sun, Yong Li, Chen Zhang, and Tao Xie. VecAttention: Vector- wise sparse attention for accelerating long context inference. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026

    2026

  45. [45]

    QUEST: Query-aware sparsity for efficient long-context LLM inference

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. QUEST: Query-aware sparsity for efficient long-context LLM inference. InProceedings of the 41st International Conference on Machine Learning (ICML), 2024

    2024

  46. [46]

    A training-free sub-quadratic cost transformer model serving framework with hierarchically pruned attention

    Heejun Lee, Geon Park, Youngwan Lee, Jaduk Suh, Jina Kim, Wonyoung Jeong, Bumsik Kim, Hyemin Lee, Myeongjae Jeon, and Sung Ju Hwang. A training-free sub-quadratic cost transformer model serving framework with hierarchically pruned attention. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  47. [47]

    FlexPrefill: A context-aware sparse attention mechanism for efficient long-sequence inference

    Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. FlexPrefill: A context-aware sparse attention mechanism for efficient long-sequence inference. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  48. [48]

    Gonzalez, and Ion Stoica

    Aditya Desai, Shuo Yang, Alejandro Cuadron, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. HashAttention: Semantic sparsity for faster inference. InProceedings of the 42nd International Conference on Machine Learning (ICML), 2025

    2025

  49. [49]

    Gonzalez, Kurt Keutzer, Ion Stoica, and Alvin Cheung

    Xuanyi Zhou, Qiuyang Mang, Shuo Yang, Haocheng Xi, Jintao Zhang, Huanzhi Mao, Joseph E. Gonzalez, Kurt Keutzer, Ion Stoica, and Alvin Cheung. SVG-EAR: Parameter-free linear compensation for sparse video generation via error-aware routing. arXiv preprint arXiv:2603.08982, 2026

    Pith/arXiv arXiv 2026

  50. [50]

    Efficient long-context modeling in diffusion language models via block approximate sparse attention.arXiv preprint arXiv:2605.19726, 2026

    Wenhu Zhang, Yiming Wu, Huanyu Wang, Yaoyang Liu, Huanzhang Dou, Senqiao Yang, Sitong Wu, Hanbin Zhao, and Jiaya Jia. Efficient long-context modeling in diffusion language models via block approximate sparse attention.arXiv preprint arXiv:2605.19726, 2026

    Pith/arXiv arXiv 2026

  51. [51]

    Gonzalez, Jianfei Chen, and Jun Zhu

    Jintao Zhang, Haoxu Wang, Kai Jiang, Shuo Yang, Kaiwen Zheng, Haocheng Xi, Ziteng Wang, Hongzhou Zhu, Min Zhao, Ion Stoica, Joseph E. Gonzalez, Jianfei Chen, and Jun Zhu. SLA: Beyond sparsity in diffusion transformers via fine-tunable sparse-linear attention. InInternational Conference on Learning Representations (ICLR), 2026

    2026

  52. [52]

    Gonzalez

    Jintao Zhang, Haoxu Wang, Kai Jiang, Kaiwen Zheng, Youhe Jiang, Ion Stoica, Jianfei Chen, Jun Zhu, and Joseph E. Gonzalez. SLA2: Sparse-linear attention with learnable routing and QAT.arXiv preprint arXiv:2602.12675, 2026. 16

    arXiv 2026