Pith. sign in

REVIEW 4 major objections 5 minor 36 references

NABLA: Neighborhood Adaptive Block-Level Attention

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a dynamic block-level sparse attention mask built by average-pooling queries and keys and thresholding the cumulative distribution of the reduced attention map can replace full self-attention in video diffusion…

desk verdict Practical block-level sparse attention with real speedups, but the average-pooling proxy at its core is unvalidated and the closest dynamic baselines are missing. read the letter →

arxiv 2507.13546 v2 pith:F62HD43Z submitted 2025-07-17 cs.CV

Dmitrii Mikhailov , Aleksey Letunovskiy , Maria Kovaleva , Vladimir Arkhipkin , Vladimir Korviakov , Vladimir Polovnikov , Viacheslav Vasilev , Evelina Sidorova
show 1 more author
Denis Dimitrov
This is my paper
classification cs.CV
keywords efficientattentionsparsevideodiffusiontransformersblock-levelsparsityCDFthresholdingFlexgenerationmodelpretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that video diffusion transformers do not need full self-attention: a per-head sparse mask computed cheaply from downsampled queries and keys can decide where attention is computed and preserve generation quality. NABLA builds the mask by average-pooling queries and keys into blocks, applying softmax to the reduced attention map, and thresholding each row's cumulative distribution so that only the blocks covering a fixed probability mass are kept. If the claim holds, high-resolution and long-duration video generation becomes substantially cheaper: the paper reports a 2.7x inference speedup on a 14B text-to-video model at 720p with CLIP, FVD, and VBench scores essentially unchanged, and a 1.46x pretraining speedup on a 2B diffusion transformer at 512x512 with lower validation loss. The result would matter because it points toward sparse attention as a default for training as well as inference, without custom kernels or profiling stages.

What carries the argument

The load-bearing object is block-level CDF sparsification of a reduced attention map. Algorithm 1 reshapes queries and keys into N-token blocks, averages each block into one query and one key vector, computes softmax over the resulting reduced map, sorts each row, forms the cumulative sum, and binarizes all entries at or after the point where the cumulative mass reaches 1 - thr. The resulting binary mask tells the FlexAttention operator which N by N blocks to compute. The CDF choice makes the number of kept blocks adapt per head and per query row: sharply peaked rows keep few blocks, while diffuse rows keep many. The method then optionally ORs this mask with a sliding-tile attention mask, giving a static locality prior that suppresses boundary artifacts. Appendix C adds a geometric-decay model in which CDF binarization has strictly lower expected L1 error than fixed Top-k at the same average block budget.

What would settle it

Take a layer of Wan 2.1, compute the full per-token attention map, aggregate it into the same N by N blocks, and compare NABLA's CDF block mask with the mask of the blocks that actually hold the highest true block-level probability mass at the same sparsity budget; if the two masks disagree often, or if the dropped blocks account for more than a few percent of a row's true probability mass, the assertion that block-level CDF sparsification preserves quality is refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a block-level mask derived from the cumulative distribution function of a downsampled attention map is a sufficient sparsity oracle for video diffusion transformers. Instead of committing to a fixed window, NABLA computes, per head, an S/N by S/N softmax map from average-pooled queries and keys, then for each row keeps the smallest prefix of sorted entries whose cumulative mass reaches 1 - thr, and binarizes that prefix into the block mask. The reported consequence is full quality recovery: on Wan 2.1 14B at 720p, CLIP rises from 42.06 to 42.08, FVD improves from 68.9 to 67.5, and VBench total moves from 83.16 to 83.17 at about 80-92 percent sparsity, while static sliding-tile attention at comparable sparsity loses semantic fidelity. The pretraining experiment extends the claim from inference to training: a 2B DiT at 512x512 with 80 percent sparse NABLA attention reaches lower validation loss per iteration than full attention and runs 10.9 seconds versus 7.5 seconds per iteration.

Load-bearing premise

The quality-preservation claim rests on the assumption that the average-pooled query and key of a token block faithfully indicate which blocks hold the significant attention mass; a block whose importance is carried by a few salient tokens could be dropped even though the reduced attention map gives it low weight.

Editorial extensions

If this is right

  • At roughly 80 percent sparsity on Wan 2.1 14B, NABLA matches the baseline's CLIP, FVD, and VBench totals within reported noise, and the side-by-side human evaluation finds no statistically significant preference between baseline and NABLA.
  • At 90 percent-plus sparsity, NABLA retains quality metrics where a static sliding-tile mask at similar sparsity degrades, especially on multiple-object and spatial-relationship scores.
  • NABLA is trainable in the strict sense: a 2B DiT pretrained at 512x512 with 80 percent sparsity converges to lower validation loss than the full-attention model while each iteration drops from 10.9 seconds to 7.5 seconds.
  • Because the mask is produced by ordinary pooling, sorting, and cumulative sums executed inside the PyTorch graph, no custom CUDA kernel or auxiliary loss is required, making the method portable across GPU hardware and compatible with existing DiT training loops.
  • The method generalizes across resolutions and aspect ratios without hyperparameter retuning, because the pooling and thresholding operate on fixed-size local blocks rather than on the global sequence length.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not explore NABLA outside video DiTs; if the average-pooled block mask preserves attention-mass location, the same three-step recipe of block pooling, CDF binarization, and static-prior union could transfer to long-context language-model attention, image DiTs, or autoregressive video models where locality holds after reordering.
  • The geometric-decay theorem compares CDF thresholding with Top-k at equal average retained blocks; a direct corollary the paper leaves implicit is that the speedup itself is content-dependent, since diffuse heads force a larger retained prefix. A practical extension would track per-head retained-block counts across timesteps and set the threshold dynamically per layer.
  • The pretraining result of lower validation loss with 80 percent sparsity hints that sparsification may act as an inductive bias or regularizer rather than a pure approximation; the paper does not claim this, but it is testable by sweeping the threshold during pretraining and checking whether any sparsity level beats full attention.
  • Because NABLA unions with sliding-tile attention, the framework suggests a general design: any cheap static prior, temporal, causal, or multiscale, can be OR-ed into the learned adaptive mask to fix the artifacts that prior addresses; exploring priors other than sliding tiles is an untested direction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes NABLA, a block-level sparse attention mechanism for video diffusion transformers. The method computes a per-head binary mask by average-pooling queries and keys into N-token blocks, softmaxing the resulting reduced attention map, and retaining blocks whose cumulative probability mass reaches a threshold 1-thr; the mask is optionally unioned with Sliding Tile Attention to avoid boundary artifacts. The mask is used with FlexAttention, requiring no custom kernels. Experiments on fine-tuning Wan 2.1 14B at 720p report up to 2.7x inference speedup with near-identical CLIP, FVD, and VBench scores, and a 1.46x pretraining speedup on a 2B DiT with lower validation loss. A proof in Appendix C claims CDF-based sparsification has lower expected L1 error than fixed Top-k under a geometric-decay model.

Significance. If the empirical claims hold, NABLA is practically valuable: it is simple, hardware-portable through FlexAttention, and appears to preserve generation quality at high sparsity, including on long-range and multi-object prompts where static STA degrades. The paper ships a human evaluation with confidence intervals and a pretraining comparison, and the method is plausibly useful for both inference and training. However, the central validity of the block-importance proxy is not directly established, and the theoretical justification in Appendix C contains a flawed proof, tempering the strength of the contributions.

major comments (4)
  1. [Algorithm 1, Section III-A2] The central quality-preservation claim rests on the assumption that average-pooled queries and keys identify the blocks that contain the significant token-level attention mass. This assumption is not directly tested. Please add a per-block fidelity analysis: for a sample of prompts, timesteps, heads, and layers, compute the full teacher attention matrix, aggregate token-level attention mass into N×N blocks, and measure the recall of true block mass inside the blocks retained by NABLA (and by NABLA∪STA). Report the fraction of retained true mass and, where possible, compare against an oracle block selector. This would resolve the concern that blocks containing a few highly salient tokens are dropped, which the current point-estimate parity does not rule out.
  2. [Appendix C, Theorem 1 and Proof] The proof contains an invalid inference: from E[ρ_i^k] ≤ τ it is claimed that ρ_i^k ≤ τ almost surely, otherwise the expectation would exceed τ. This is false when the distribution of ρ_i has mass below τ. In fact, the theorem as stated is not true. For example, with k=2 and ρ uniform on [0.85,0.95], the threshold τ satisfying E[r_i]=2 yields E[ε_cdf] > E[ε_top], contradicting the claimed strict inequality. Additionally, the assertion that τ↦E[r_i(τ)] is strictly decreasing and continuous is generally false because r_i is integer-valued, giving step-function behavior and non-uniqueness of τ. The theorem should be corrected or removed; the empirical CDF heuristic may still be reasonable, but the stated mathematical guarantee is not.
  3. [Tables 3 and 4] The objective quality metrics (CLIP, FVD, VBench) are reported as single point estimates without error bars, confidence intervals, or significance tests. Given that the claimed parity involves differences as small as 0.02 in CLIP and 0.01 in VBench, the reader cannot assess whether these differences are meaningful. Please report bootstrap CIs, multiple evaluation runs, or paired significance tests. The human evaluation in Table 5 partially addresses subjective parity, but objective metrics should be accompanied by uncertainty estimates.
  4. [Section IV-B, Table 5] The human evaluation uses 50 participants and 20 video pairs each, but the table caption states estimated N=500, which appears to assume all participants rated all pairs against the same method pair. Please clarify the total number of pairwise judgments per condition and the exact composition of the 20 video pairs across the configurations being compared, since the reported confidence intervals are only meaningful if the sample structure is correctly described.
minor comments (5)
  1. [Abstract] The abstract states that NABLA achieves 'up to 2.7x faster training and inference', but the reported training speedup is 1.46x and the inference speedup is 2.7x. Please rephrase to state the two numbers separately to avoid overclaiming.
  2. [Algorithm 1] The notation 'sort(A)' in step 7 is ambiguous; specify that sorting and cumsum are applied along the key-block dimension, and clarify how 'reorder' in step 10 restores the original block order.
  3. [Table 2] The sparsity percentages for the compared configurations differ slightly (e.g., 80.13% vs 79.45%), but the inference times are reported for those specific configurations. Please state whether sparsity is measured on the same evaluation prompts or is an average over the dataset, and whether the threshold thr is tuned separately for each configuration.
  4. [Section IV-C] The pretraining experiment reports lower validation loss for NABLA but does not describe the validation set, the sparsity schedule during training, or whether the STA union is used. Adding these details would make the pretraining claim reproducible.
  5. [References] Reference [16] (DSV) is cited as arXiv:2502.07590, 2024, but the arXiv listing is 2025; please verify the year and, if applicable, the venue.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: NABLA's mask construction and quality claims are measured against full attention, and Appendix C is a standalone theoretical motivation rather than a derivation of the experimental results.

full rationale

The paper contains no circular derivation. The NABLA mask is constructed from average-pooled queries and keys via CDF thresholding, and the quality-preservation claim is established by external metrics (CLIP, FVD, VBench, human evaluation) comparing NABLA-finetuned models against the full-attention baseline. The threshold parameter `thr` is a tuned hyperparameter per configuration, not a fitted parameter that emulates the target metrics. Appendix C proves only that CDF-based sparsification has lower expected L1 error than fixed Top-k under an explicit geometric-decay model of token-level attention rows; this theorem is presented as motivation, with the paper itself stating in Appendix C.D that 'the practical benefit depends on implementation overhead' and that 'its superiority must be validated empirically.' References [13] and [14] are self-citations to the authors' prior Kandinsky models, but they appear only as examples in a list of video generation systems and are not load-bearing for any claim about NABLA. The average-pooling proxy for block importance is an internal-validity concern, not a circularity, because the selected blocks are compared with the actual full-attention output only through downstream quality metrics, and no equation in the paper defines those metrics in terms of the mask-generation inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method depends on tuned hyperparameters (threshold, block size, STA window, patch size) and on assumptions that average-pooled QK attention approximates true block importance and that attention rows decay geometrically. No new physical or architectural entities are postulated.

free parameters (4)
  • thr (binarization threshold) = 0.7, 0.4, 0.2 used in experiments
    Controls how much probability mass must be covered before a block is retained; directly sets sparsity and is tuned per configuration.
  • block size N = not reported for experiments
    Number of tokens averaged into one block in Algorithm 1; determines mask granularity and the cost of the reduced attention map.
  • STA window sizes (W_T, W_H, W_W) = (11,40,40) and (18,24,24)
    Fixed 3D window for the optional STA union; chosen by the authors, no automatic selection.
  • spatial patch size P = 8 shown in Figure 4, experimental value not stated
    Size of spatial patches in fractal flattening; affects token ordering and hence block structure.
assumptions (4)
  • ad hoc to paper Attention rows follow a geometric decay model: a_i(pi_i(j))=(1-rho_i)rho_i^(j-1)
    Appendix C, Eq. 6; needed for Theorem 1 that CDF thresholding beats Top-k, but the authors themselves note real attention can be multimodal or long-tailed.
  • domain assumption Average-pooled queries and keys preserve block-level attention importance
    Algorithm 1 lines 1-6; the mask is built from pooled Q and K, so the method assumes pooling does not hide important blocks.
  • domain assumption Reduced softmax attention approximates the aggregation of full attention over blocks
    Same algorithm; no theoretical bound links the pooled attention matrix to the true block-wise attention mass.
  • standard math PyTorch FlexAttention correctly applies arbitrary boolean masks
    Relied on for all speedups; not verified in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NABLA: Neighborhood Adaptive Block-Level Attention." pith.science (2026). https://pith.science/paper/F62HD43Z

@misc{pith2026250713546,
  author       = {Pith},
  title        = {Pith review of: NABLA: Neighborhood Adaptive Block-Level Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F62HD43Z}},
  note         = {Machine review of arXiv:2507.13546}
}
read the original abstract

Recent progress in transformer-based architectures has demonstrated remarkable success in video generation tasks. However, the quadratic complexity of full attention mechanisms remains a critical bottleneck, particularly for high-resolution and long-duration video sequences. In this paper, we propose NABLA, a novel Neighborhood Adaptive Block-Level Attention mechanism that dynamically adapts to sparsity patterns in video diffusion transformers (DiTs). By leveraging block-wise attention with adaptive sparsity-driven threshold, NABLA reduces computational overhead while preserving generative quality. Our method does not require custom low-level operator design and can be seamlessly integrated with PyTorch's Flex Attention operator. Experiments demonstrate that NABLA achieves up to 2.7x faster training and inference compared to baseline almost without compromising quantitative metrics (CLIP score, VBench score, human evaluation score) and visual quality drop. The code and model weights are available here: https://github.com/gen-ai-team/Wan2.1-NABLA

Figures

Figures reproduced from arXiv: 2507.13546 by the authors.

Figure 1
Figure 1. FIGURE 1: The block-sparse attention mask is computed [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 1
Figure 1. Extensive evaluations on video datasets demon [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. FIGURE 2: Examples of attention maps (left) and corresponding NABLA masks (right) for different heads of Wan 2.1 14B [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: FIGURE 4: Token reordering illustration for a la [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: FIGURE 5: Training convergence for full attention and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: FIGURE 7: Input prompt [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: FIGURE 8: Input prompt [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 6
Figure 6. Figure 6: FIGURE 6: Input prompt [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 9
Figure 9. Figure 9: FIGURE 9: Input prompt [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 28 canonical work pages

  1. [1]

    J. Ho, A. Jain, and P . Abbeel, ‘‘Denoising diffusion probabilistic models,’’ inAdvances in Neural Information Processing Systems(H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, eds.), vol. 33, pp. 6840– 6851, Curran Associates, Inc., 2020

    work page 2020

  2. [2]

    Peebles and S

    W. Peebles and S. Xie, ‘‘Scalable diffusion models with transformers,’’ arXiv preprint arXiv:2212.09748, 2022

    arXiv 2022

  3. [3]

    Rombach, A

    R. Rombach, A. Blattmann, D. Lorenz, P . Esser, and B. Ommer, ‘‘High- resolution image synthesis with latent diffusion models,’’ 2021

    work page 2021

  4. [4]

    D. P . Kingma and M. Welling, ‘‘Auto-Encoding V ariational Bayes,’’ in2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014

    work page 2014

  5. [5]

    Esser, S

    P . Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y . Levi, D. Lorenz, A. Sauer, F. Boesel, D. Podell, T. Dockhorn, Z. English, and R. Rombach, ‘‘Scaling rectified flow transformers for high-resolution image synthesis,’’ inProceedings of the 41st International Conference on Machine Learning, ICML’24, JMLR.org, 2024

    work page 2024

  6. [6]

    V aswaniet al., ‘‘Attention is all you need,’’Advances in neural infor- mation processing systems, vol

    A. V aswaniet al., ‘‘Attention is all you need,’’Advances in neural infor- mation processing systems, vol. 30, 2017

    work page 2017

  7. [7]

    com/index/video-generation-models-as-world-simulators/, 2024

    OpenAI, ‘‘Video generation models as world simulators.’’ https://openai. com/index/video-generation-models-as-world-simulators/, 2024

    work page 2024

  8. [8]

    Polyak, A

    A. Polyak, A. Zohar, A. Brown, A. Tjandra, A. Sinha, A. Lee, A. Vyas, B. Shi, C.-Y . Ma, C.-Y . Chuang, D. Y an, D. Choudhary, D. Wang, G. Sethi, G. Pang, H. Ma, I. Misra, J. Hou, J. Wang, K. Jagadeesh, K. Li, L. Zhang, M. Singh, M. Williamson, M. Le, M. Y u, M. K. Singh, P . Zhang, P . V ajda, Q. Duval, R. Girdhar, R. Sumbaly, S. S. Rambhatla, S. Tsai, S...

    work page 2025

Show all 36 references
  1. [9]

    W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, K. Wu, Q. Lin, J. Y uan, Y . Long, A. Wang, A. Wang, C. Li, D. Huang, F. Y ang, H. Tan, H. Wang, J. Song, J. Bai, J. Wu, J. Xue, J. Wang, K. Wang, M. Liu, P . Li, S. Li, W. Wang, W. Y u, X. ...

    2025

  2. [10]

    Y ang, J

    Z. Y ang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y . Y ang, W. Hong, X. Zhang, G. Feng, D. Yin, Y . Zhang, W. Wang, Y . Cheng, B. Xu, X. Gu, Y . Dong, and J. Tang, ‘‘Cogvideox: Text-to-video diffusion models with an expert transformer,’’ 2025

    2025

  3. [11]

    AI, ‘‘Kling ai: Next-gen ai video & ai image generator.’’ https://app

    K. AI, ‘‘Kling ai: Next-gen ai video & ai image generator.’’ https://app. klingai.com/global/, 2024

    2024

  4. [12]

    T. Wan, A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Y u, H. Zhao, J. Y ang, J. Zeng, J. Wang, J. Zhang, J. Zhou, J. Wang, J. Chen, K. Zhu, K. Zhao, K. Y an, L. Huang, M. Feng, N. Zhang, P . Li, P . Wu, R. Chu, R. Feng, S. Zhang, S. Sun, T. Fang, T. Wang, T. Gui, T. ...

    2025 arXiv

  5. [13]

    Arkhipkin, Z

    V . Arkhipkin, Z. Shaheen, V . V asilev, E. Dakhova, K. Sobolev, A. Kuznetsov, and D. Dimitrov, ‘‘Improveyourvideos: Architectural im- provements for text-to-video generation pipeline,’’IEEE Access, vol. 13, pp. 1986–2003, 2025

    1986

  6. [14]

    Arkhipkin, V

    V . Arkhipkin, V . V asilev, A. Filatov, I. Pavlov, J. Agafonova, N. Gerasimenko, A. Averchenkova, E. Mironova, B. Anton, K. Kulikov, A. Kuznetsov, and D. Dimitrov, ‘‘Kandinsky 3: Text-to-image synthesis for multifunctional generative framework,’’ inProceedings of the 2024 Con...

    2024

  7. [15]

    Y . Deng, Z. Song, J. Xiong, and C. Y ang, ‘‘How sparse attention approxi- mates exact attention? your attention is naturallyn c-sparse,’’ 2025

    2025

  8. [16]

    X. Tan, Y . Chen, Y . Jiang,et al., ‘‘Dsv: Exploiting dynamic sparsity to ac- celerate large-scale video dit training,’’arXiv preprint arXiv:2502.07590, 2024

    2024

  9. [17]

    Y . Xia, S. Ling, F. Fu, Y . Wang, H. Li, X. Xiao, and B. Cui, ‘‘Training-free and adaptive sparse attention for efficient long video generation,’’ 2025

    2025

  10. [18]

    Zhang, C

    J. Zhang, C. Xiang, H. Huang, J. Wei, H. Xi, J. Zhu, and J. Chen, ‘‘Spargeattn: Accurate sparse attention accelerating any model inference,’’ 2025

    2025

  11. [19]

    Jiang, Y

    H. Jiang, Y . Li, C. Zhang,et al., ‘‘Minference 1.0: Accelerating pre- filling for long-context llms via dynamic sparse attention,’’arXiv preprint arXiv:2407.02490, 2024

    2024 arXiv

  12. [20]

    Beltagy, M

    I. Beltagy, M. E. Peters, and A. Cohan, ‘‘Longformer: The long-document transformer,’’arXiv:2004.05150, 2020

    2004 arXiv

  13. [21]

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, ‘‘Swin transformer: Hierarchical vision transformer using shifted windows,’’ in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021

    2021

  14. [22]

    Hassani, S

    A. Hassani, S. Walton, J. Li, S. Li, and H. Shi, ‘‘Neighborhood attention transformer,’’ inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6185–6194, 6 2023

    2023

  15. [23]

    Hassani and H

    A. Hassani and H. Shi, ‘‘Dilated neighborhood attention transformer,’’ arXiv:2209.15001, 2022

    2022 arXiv

  16. [24]

    Hassani, W.-M

    A. Hassani, W.-M. Hwu, and H. Shi, ‘‘Faster neighborhood attention: Reducing theo(n 2)cost of self attention at the threadblock level,’’ in Advances in Neural Information Processing Systems, 2024

    2024

  17. [25]

    Zhang, Y

    P . Zhang, Y . Chen, R. Su, H. Ding, I. Stoica, Z. Liu, and H. Zhang, ‘‘Fast video generation with sliding tile attention,’’ 2025

    2025

  18. [26]

    H. Xi, S. Y ang, Y . Zhao,et al., ‘‘Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity,’’arXiv preprint arXiv:2502.01776, 2025

    2025 arXiv

  19. [27]

    Y . Wen, J. Wu, A. Jain, T. Goldstein, and A. Panda, ‘‘Analysis of attention in video diffusion transformers,’’ 2025

    2025

  20. [28]

    Pagliardini, D

    M. Pagliardini, D. Paliotta, M. Jaggi, and F. Fleuret, ‘‘Fast attention over long sequences with dynamic sparse flash attention,’’ inAdvances in Neu- ral Information Processing Systems(A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, eds.), vol. 36, pp. 598...

    2023

  21. [29]

    Kitaev, Łukasz Kaiser, and A

    N. Kitaev, Łukasz Kaiser, and A. Levskaya, ‘‘Reformer: The efficient transformer,’’ 2020

    2020

  22. [30]

    S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma, ‘‘Linformer: Self- attention with linear complexity,’’ 2020

    2020

  23. [31]

    Gonçalves, M

    N. Gonçalves, M. Treviso, and A. F. T. Martins, ‘‘Adasplash: Adaptive sparse flash attention,’’arXiv preprint arXiv:2502.12082, 2025

    2025 arXiv

  24. [32]

    Willette, H

    J. Willette, H. Lee, and S. J. Hwang, ‘‘Delta attention: Fast and accurate sparse attention inference by delta correction,’’ 2025

    2025

  25. [33]

    Y uan, H

    J. Y uan, H. Gao, D. Dai, J. Luo, L. Zhao, Z. Zhang, Z. Xie, Y . X. Wei, L. Wang, Z. Xiao, Y . Wang, C. Ruan, M. Zhang, W. Liang, and W. Zeng, ‘‘Native sparse attention: Hardware-aligned and natively trainable sparse attention,’’ArXiv, vol. abs/2502.11089, 2025

    2025 arXiv

  26. [34]

    H. Zhou, J. Tang, J. Zhang, Y . Li, C. Xiao, L. Hou, Z. Ke, and J. Y ao, ‘‘Comem: Compositional concept-graph memory for vision–language adaptation,’’ inInternational Conference on Learning Representations (ICLR), 2026

    2026

  27. [35]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Un- terthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, ‘‘An image is worth 16x16 words: Transformers for image recognition at scale,’’ 2021. 8

    2021

  28. [36]

    J. Dong, B. Feng, D. Guessous, Y . Liang, and H. He, ‘‘Flex attention: A programming model for generating optimized attention kernels,’’ 2024. APPENDIX A FINE-TUNING HYPERPARAMETERS We performed knowledge distillation on the full-attention Wan2.1 open-source model using MSE lo...

    2007

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.