Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

This paper claims that skipping attention scores as Mx1 slices rather than MxM blocks makes video diffusion generation over 1.5x faster with negligible quality loss.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 16:03 UTC pith:ZXPWT5DD

load-bearing objection Genuinely new fine-grained sparse attention kernel with plausible video-DiT speedups, but the quality claim rests on an under-tested heuristic and the numbers need reconciling. the 4 major comments →

arxiv 2509.16518 v2 pith:ZXPWT5DD submitted 2025-09-20 cs.CV cs.AR

FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Models

Sankeerth Durvasula , Kavya Sreedhar , Zain Moustafa , Suraj Kothawade , Tianlei Pang , Ashish Gondimalla , Suvinay Subramanian , Narges Shahidi
show 1 more author
Nandita Vijaykumar
This is my paper
classification cs.CV cs.AR
keywords sparse attentionvideo diffusion transformerinference accelerationGPU kernelattention maskfine-grained sparsitytensor coresdiffusion model inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's central claim is that attention maps in video diffusion transformers are sparse at a much finer grain than current coarse block-sparse methods assume, and that this sparsity can be harvested for real speedups. It proposes FG-Attn, which skips attention-score computation per Mx1 slice—a block of M contiguous queries evaluated against a single key—instead of per MxM block. To keep the kernel efficient, it introduces an asynchronous gather-load primitive that fetches only the relevant sparse key/value vectors into shared memory and packs them into tensor-core-compatible tiles, hiding address-generation latency behind attention computation. Two training-free mask strategies decide which slices to keep: caching the mask from an earlier denoising iteration, and thresholding dot products of a group's mean query. On state-of-the-art video models, FG-Attn reports an average 1.48x end-to-end speedup, up to 1.65x, with essentially unchanged aggregate quality.

Core claim

FG-Attn establishes that the attention-score matrices of video diffusion transformers contain abundant fine-grained sparsity: dropping to 16x16 blocks reveals roughly 70% of scores below threshold, versus only about 22% at 64x64 blocks, yet generating with only 20% of scores still yields valid videos. The method exploits this by treating each Mx1 slice (M=128 in the H100 implementation) as the unit of skipping, and by loading only the keys and values corresponding to the kept slices. The load is implemented with a new asynchronous gather-load primitive that computes addresses for sparse vector indices in parallel threads and issues asynchronous memory loads, so the overhead does not sit on t

What carries the argument

The central object is the Mx1 slice of the attention-score matrix: one column of scores produced by M contiguous queries against a single key. FG-Attn decides at this granularity which slices to compute, skipping 64x64 or 128x128 blocks entirely and instead keeping only the slices that the mask marks as significant. The enabling mechanism is the asynchronous gather-load primitive, which takes the sparse list of key/value indices, distributes address generation across threads and warps, and loads exactly those vectors into shared memory in a swizzled, tensor-core-ready tile while the compute side of the kernel proceeds in parallel. The mask itself comes from two training-free procedures: a ca

Load-bearing premise

The load-bearing premise is that the set of significant attention slices for a group of queries can be reliably predicted from the group's mean query (or from the previous denoising iteration); if queries within a group attend to genuinely different keys, or patterns shift between iterations, the mask drops essential scores and quality suffers.

What would settle it

Take a set of prompts, compute full attention maps, mark every Mx1 slice containing at least one score above the threshold used by FG-Attn, and compare that ground-truth slice set with the slices selected by FG-Attn-cached and FG-Attn-pooling; if the predicted mask's recall is well below 100% on typical prompts, the claimed quality preservation is not supported. Additionally, run FG-Attn-pooling over a prompt set and check per-prompt quality-metric variance against the baseline—large variance would show the mean-query heuristic is prompt-dependent.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Video diffusion attention is far sparser than block-sparse methods exploit; switching from 64x64 blocks to 16x16 or Mx1 granularity exposes roughly 3x more skippable computation on typical attention maps.
  • FG-Attn delivers average 1.48x (up to 1.65x) end-to-end video generation speedup on single-H100 runs of Wan 2.1 1.3B/14B and HunyuanVideo, and the attention kernel itself is about 2x faster than the PyTorch baseline.
  • Because the mask can be cached across denoising iterations, the one-time cost of computing a full attention map is amortized over the remaining iterations, making the sparse kernel nearly free of masking overhead.
  • FG-Attn's slice-based skipping is a drop-in replacement for block-sparse kernels: it matches or exceeds the speed of prior coarse-grained sparse attention methods on the same models, and learned or static masks from those methods can be fed directly into FG-Attn.
  • The method is training-free, so it can be applied to existing video diffusion models without fine-tuning or retraining.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the reported slice-level sparsity and cross-iteration stability generalize, the same Mx1 skipping plus gather-load recipe should transfer to other long-context diffusion workloads (3D, audio, image) and to LLM prefill, though the paper itself flags that memory-bound decoding may not hide the gather latency.
  • The average-query heuristic makes a testable locality assumption: adjacent spatial-temporal embeddings have similar attention patterns. A direct measurement of the divergence between top-k keys per mean query and top-k keys per individual query would give a sharper bound on when FG-Attn-pooling degrades than aggregate quality scores do.
  • Mask recall—how many truly significant slices the cached or pooled mask retains—would be a stronger quality gate than VBench aggregates; per-prompt quality variance would reveal whether threshold choices work uniformly across prompts or only on average.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes FG-Attn, a fine-grained sparse attention mechanism for video diffusion transformers. Instead of skipping entire M×M blocks of attention scores as in block-sparse attention, FG-Attn skips M×1 slices (M≥16, N=1 in the tile), where a slice corresponds to M queries and a single key. To make this efficient on H100 GPUs, the authors introduce an asynchronous gather-load primitive that packs sparse key/value vectors into tensor-core-compatible tiles in shared memory. Two training-free mask-generation strategies are proposed: a caching strategy that reuses the significant-slice mask across denoising iterations, and a pooling strategy that uses the average query of a group to score keys. The paper reports end-to-end speedups on Wan 2.1 and HunyuanVideo, with VBench quality metrics showing negligible degradation, and claims that FG-Attn can supersede existing block-sparse attention methods.

Significance. If the reported results hold, FG-Attn addresses a real bottleneck: attention computation dominates video DiT inference, and existing 64×64 block-sparse methods leave much of the fine-grained sparsity unexploited. The M×1 slice granularity, combined with the asynchronous gather-load primitive, is a plausible and interesting systems contribution, and the open-source code link is a strength. The paper also provides a useful quantitative analysis of attention-map sparsity at different block sizes (Table 2). However, the central quantitative claim is undermined by inconsistent speedup numbers across the abstract, the body, and the conclusions, as well as by the absence of the claimed FlashInfer and FlashAttention-3 comparisons in the experimental section. The quality-preservation claim also rests on the average-query heuristic whose threshold is undefined and whose recall is never measured. These issues are fixable within the scope of the paper, so the work warrants revision rather than rejection.

major comments (4)
  1. [Abstract (metadata), §5.2, §8] The speedup numbers are internally inconsistent: the metadata abstract reports up to 1.40×/1.18× average over FlashAttention 3; the full-text abstract reports average 1.55×/1.41× for 480p/720p; the body (§5.2) reports an average 1.48× and up to 1.65×; and Fig. 12 reports attention-kernel speedups of 1.93× average and 2.38× max. The experiments in Section 5 compare against a PyTorch baseline, SparseVideoGen, and RadialAttention, but not against FlashAttention 3 or FlashInfer, both of which appear in the abstract/related work. This makes the central speedup claim irreproducible. Please reconcile all reported numbers and provide a table of raw end-to-end times with clearly stated baselines.
  2. [§4.4 and Table 3] The average-query masking strategy is not sufficiently validated. The threshold in Eq. (the formula with exp(k·qavg/√D)/D < τ1) is undefined: τ1 is never specified in the text or in Section 5.1 (the cached threshold is given as 0.5/N, but the pooling threshold is not). More importantly, the quality claim 'negligible degradation' is supported only by aggregate VBench scores in Table 3 and three visual examples. No error bars are reported, and per-prompt failures can be hidden by aggregate scores. Since the pooling mask is derived from qavg rather than from each individual query, a key that is important for one query in the group may be skipped if it scores low against qavg. Please report mask recall (fraction of significant per-query keys retained by the qavg mask) and per-prompt quality metrics (e.g., PSNR/LPIPS against the full-attention output, or a dispersion measure of VBench scores)
  3. [§4.2 and §4.5] The memory overhead of the sparse-index mask is stated as O(BHN^2/M) and dismissed as 'negligible runtime overhead,' but the absolute memory footprint is never quantified. For N≈74,000 (720p, 5s), M=128, and a typical number of heads, the cached mask [B,H,N/M,N] of 32-bit integers can be several GB per batch. This is not negligible for the caching strategy and may affect its feasibility on memory-constrained GPUs. Please report the measured HBM usage for the evaluated configurations and separate memory overhead from runtime overhead.
  4. [§5.2, Fig. 11-13] The speedup plots and ablation do not report the number of runs, variance, or confidence intervals. The normalized end-to-end times in Fig. 11 appear to be single measurements; the ablation in Fig. 13 has no error bars. Given that diffusion sampling is stochastic, run-to-run variance in generation time (due to dynamic masking and asynchronous loads) should be characterized. Please include at least 3–5 repeated runs for each configuration and report mean ± std, or justify why variance is negligible.
minor comments (6)
  1. [§4.1] Typo: 'altenative' should be 'alternative' in the description of the second mask-generation strategy.
  2. [§5.1] Typo: 'spatio templaral tokens' should be 'spatio-temporal tokens'.
  3. [§5.2, Fig. 11-12] Legend labels 'FGSpAttn-pooled' and 'FGSpAttn-cached' are inconsistent with the text's 'FG-Attn-pooling' and 'FG-Attn-cached'. Please use one consistent naming throughout.
  4. [§4.4] The notation for thresholds is inconsistent: τcached is mentioned in the text, the pooling formula uses τ1, and Section 5.1 introduces a threshold of 0.5/N for the cached strategy. Please define every threshold symbol at first use.
  5. [§1 and §3.2] The FLOP-reduction estimates for 16×16 blocks (70%) and 128×1 slices (55%) are not tied to the measured speedups. Consider adding a small table that maps sparsity, FLOP reduction, and measured kernel speedup for each configuration, so the reader can distinguish theoretical savings from achieved speedups.
  6. [References] Some references are incomplete or inconsistently formatted (e.g., 'Spargeattention' vs. 'SpargeAttention'; several arXiv entries lack version numbers). Please proofread the reference list.

Circularity Check

0 steps flagged

No significant circularity; central claims are empirical speedup and quality measurements.

full rationale

The paper's central claims are measured end-to-end and attention-kernel speedups (Section 5.2) against external baselines (FlashAttention, SparseVideoGen, Radial Attention) and VBench quality metrics (Section 5.3). The sparse mask thresholds (τ_cached = 0.5/N, and the pooling heuristic τ1) are heuristic parameters, not fitted to reproduce the reported speedups or quality numbers. The caching strategy (Section 4.4) derives masks from the model's own attention scores in a previous denoising iteration; this is a runtime optimization, not a self-definitional or fitted prediction. The pooling strategy (Section 4.4) uses qavg to select keys; while the paper does not measure mask recall and the inequality for τ1 appears ambiguous, this is a validity/correctness risk rather than circularity. The paper cites prior works (Hu et al. 2025; Ma et al. 2025) for the observation that embeddings change slowly across denoising steps; these are external works and the observation is used as motivation, not as an input that forces the reported results. No self-citation is load-bearing, no uniqueness theorem is imported, and no known result is simply renamed. The derivation chain is self-contained: the sparsity analysis (Table 2) motivates the design, but the speedup is measured empirically, not derived from a formula that includes the target outcome.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical entities. Its contributions are a kernel primitive (asynchronous gather-load) and two mask heuristics. The load-bearing premises are the two domain assumptions about query similarity and mask stability, plus the chosen thresholds, which are free parameters that directly control the speedup/quality trade-off.

free parameters (4)
  • threshold for cached mask (tau_cached) = 0.5/N
    Set in Section 5.1 to 0.5/N where N is sequence length; chosen by hand, not derived, and directly controls how many slices are skipped.
  • threshold for pooling mask (tau_1) = unspecified
    Defined in the inequality in Section 4.4 but no value is given in the experiments, making the pooling variant's behavior under-specified.
  • cache refresh interval = every 15 DiT iterations
    Chosen in Section 5.1; affects mask freshness and quality, but no sensitivity analysis is provided.
  • query block size M = 128 (for Hopper implementation)
    Implementation choice aligned with tensor core dimensions (Section 4.2); impacts granularity and overhead.
axioms (4)
  • domain assumption Most significant attention scores are concentrated in static regions, especially around the diagonal, across all layers.
    Invoked in Section 1 and Section 3.2 when motivating why a fixed or cached mask can work; sourced from prior work but assumed for this paper's method.
  • ad hoc to paper Nearby embeddings (queries) tend to produce similar query distributions, so a group's mean query represents the group.
    The central assumption of the pooling strategy in Section 4.4; not measured or validated independently, only via end-to-end VBench scores.
  • domain assumption Across denoising iterations, the intermediate embeddings remain approximately unchanged, so attention masks are stable enough to cache.
    Used in Section 4.4 for the cached-mask strategy; cites Hu et al. 2025 and Ma et al. 2025, but this paper does not quantify mask drift.
  • ad hoc to paper Skipping attention slices whose scores are below threshold does not affect output quality.
    Implicit in the entire method (Section 3.2); relies on the threshold and is validated only through the VBench table, not through a direct quality-sparsity curve for the pooling variant.

pith-pipeline@v1.3.0-alltime-deepseek · 14900 in / 9012 out tokens · 74546 ms · 2026-08-04T16:03:44.594628+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Models." pith.science (2026). https://pith.science/paper/ZXPWT5DD

@misc{pith2026250916518,
  author       = {Pith},
  title        = {Pith review of: FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZXPWT5DD}},
  note         = {Machine review of arXiv:2509.16518}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Using diffusion transformers for media generation may require evaluating attention over extremely long sequences, with attention layers accounting for the majority of generation latency. Exploiting sparsity in attention maps offers a promising opportunity to reduce this cost. In this work, we show that attention maps in diffusion transformers exhibit significant fine-grained sparsity in video generation models. Existing sparse attention methods, however, are too coarse-grained, leaving a large fraction of redundant computation unaddressed, or incur high overheads at finer granularity. We propose FG-Attn, a novel, low-overhead fine-grained sparse attention mechanism that skips score computations at the granularity of a MxN tile, where N>=1 and M>=16, and where each block is the result of query-key dot products between M queries and N keys. FG-Attn addresses the key challenge of hardware underutilization in sparse attention kernels on GPUs, without incurring the overheads of irregular memory access and redundant operations. FG-Attn can fully supersede existing sparse attention methods and extend block sparse attention methods to finer granularities on modern GPUs. At 70% sparsity, FG-Attn is up to 2.45X faster than the state-of-art FlashInfer, and reduces attention kernel time by 14.7% on average. FG-Attn speeds up end-to-end video generation times by up to 1.40X (1.18X on average) over Flash Attention 3.

Figures

Figures reproduced from arXiv: 2509.16518 by Ashish Gondimalla, Kavya Sreedhar, Nandita Vijaykumar, Narges Shahidi, Sankeerth Durvasula, Suraj Kothawade, Suvinay Subramanian, Tianlei Pang, Zain Moustafa.

Figure 1
Figure 1. Figure 1: Left: A video frame from Wan 2.1 (Wan et al., 2025) for the prompt “horse bending and drinking water from a lake”. Right: The same model generates a similar video using only 20% of attention scores per head, achieving comparable results with a fraction of the FLOPs. In this work, we aim to develop a fine-grained sparse at￾tention mechanism to accelerate diffusion model inference. Our goals are to: (1) desi… view at source ↗
Figure 2
Figure 2. Figure 2: A latent space representation of video frames, repre￾sented as a set sequence of embeddings and encoded using a VQ￾VAE, can be denoised using a diffusion transformer to produce embeddings corresponding to clean frames. Generating a video sample from the diffusion model. Producing a video corresponds to drawing a sample from the fitted score function. This can be done by solving the probability flow Ordinar… view at source ↗
Figure 4
Figure 4. Figure 4: Breakdown of time spent (in seconds) by different opera￾tions during inference of Wan 2.1 1.3B (Wan et al., 2025) (eager mode). each operator to produce the video (note that “others” here indicate the operators to encode the text tokens and the initial noisy video frames using the VQVAE). We observe that the majority of computation time is spent evaluating the transformer model, with the attention layer ac… view at source ↗
Figure 5
Figure 5. Figure 5: Sparsity in attention computation: attention scores are highly sparse, and locations of negligible attention scores are ir￾regularly distributed [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Block sparse attention mechanisms skip tiles of 128 × 128 attention map scores. We propose a method to skip fine-grain 128 × 1 sections of the attention scores. 4.2 Representing the Fine-grain Attention Mask To implement fine-grained sparse attention, we need the attention mask to specify which slices of attention map scores to compute, i.e., the key/value vectors required for every group of M queries. For… view at source ↗
Figure 7
Figure 7. Figure 7: Loading a sparse set of keys/values using the sparse index mask into a tile in shared memory. on-chip memory. We denote this relevant set of keys for a group of queries as Krel. The query tile and Krel are then supplied to the tensor core to compute pairwise query-key dot products (i.e, QtileKrel). Loading a sparse set of key/value vectors from memory involves first generating (i.e, computing) the addresse… view at source ↗
Figure 8
Figure 8. Figure 8: Pipelining gather-loading of keys/values into shared memory in the producer threads. The sparse index mask indices are loaded into registers first, followed by loading key/value tokens by the producer threads (gather-load primitive). The load latency of address generation is hidden by the attention computation. End-to-end Implementation overview. An overview of our implementation is shown in [PITH_FULL_IM… view at source ↗
Figure 10
Figure 10. Figure 10: Determining which slices to attention mask based on attention scores observed in the previous denoising iteration. sparse attention mask by evaluating the full attention scores and identifying slices with at least one score above a thresh￾old τcached. The resulting mask is then stored in HBM for each attention head. In subsequent denoising iterations, this cached mask is reused by our sparse attention mec… view at source ↗
Figure 11
Figure 11. Figure 11: shows the end-to-end time required to generate the video, normalized to baseline. We observe that FG-Attn is able to achieve an average speedup of 1.48× and up to 1.65×. FG-Attn achieves a speedup as a result of ac￾celerating the attention computation time during training [PITH_FULL_IMAGE:figures/full_fig_p009_11.png] view at source ↗
Figure 12
Figure 12. Figure 12 [PITH_FULL_IMAGE:figures/full_fig_p010_12.png] view at source ↗
Figure 14
Figure 14. Figure 14: Samples of videos generated using baseline HunyuanVideo model, and FG-Attn-HunyuanVideo (The baseline generates first row, second row generated using FG-Attn) with SparseVideoGen and Radial Attention in Section 5. Moreover, trainable sparse attention methods such as Video Sparse Attention (VSA) (Zhang et al., 2025b) can be refor￾mulated to generate sparse masks compatible with FG-Attn 's attention kernel.… view at source ↗
Figure 15
Figure 15. Figure 15: Samples of videos generated using baseline Wan-1.3B model, and FG-Attn-Wan1.3B. (First row is generated by the baseline, second row is generated using FG-Attn) it a general drop-in replacement for block-sparse attention. Our current design of FG-Attn is implemented in NVIDIA H100 GPU. However, the asynchronous gather-load prim￾itive can be re-implemented to align with other accelera￾tors’ memory hierarchi… view at source ↗
Figure 16
Figure 16. Figure 16: Samples of videos generated using baseline Wan-14B model, and FG-Attn-Wan14B (First row is generated by the baseline, second row is generated by FG-Attn) based on caching the mask across denoising iterations and another lightweight method using a query-averaging heuris￾tic. Using FG-Attn, we demonstrate an average speedup of 1.48×, and up to 1.65× on state of art video diffusion models. REFERENCES Chen, J… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

    cs.CV 2026-05 unverdicted novelty 7.0

    DFSAttn is a training-free framework for dynamic fine-grained sparse attention in video DiTs that achieves up to 2.1x speedup while preserving generation quality via Hilbert reordering, hierarchical scoring, and adapt...

  2. Efficient Video Diffusion Models: Advancements and Challenges

    cs.CV 2026-04 unverdicted novelty 7.0

    A survey that groups efficient video diffusion methods into four paradigms—step distillation, efficient attention, model compression, and cache/trajectory optimization—and outlines open challenges for practical use.

  3. Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithms

    eess.IV 2026-03 unverdicted novelty 6.0

    Video generation models can function as world simulators if efficiency gaps in spatiotemporal modeling are bridged via organized paradigms, architectures, and algorithms.

  4. Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithms

    eess.IV 2026-03 conditional novelty 5.0

    In twisted bilayer nodal d-wave superconductors, interlayer hopping creates nodes on the C2 axis and Bogoliubov flat bands when the single-layer Berry connection is parallel to that axis.

Reference graph

Works this paper leans on

18 extracted references · 15 linked inside Pith · cited by 3 Pith papers

  1. [1]

    Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis.arXiv preprint arXiv:2310.00426,

    Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y ., Wang, Z., Kwok, J., Luo, P., Lu, H., et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis.arXiv preprint arXiv:2310.00426,

    Pith/arXiv arXiv

  2. [4]

    Flashdecoding++: Faster large language model inference on gpus.arXiv preprint arXiv:2311.01282,

    Hong, K., Dai, G., Xu, J., Mao, Q., Li, X., Liu, J., Chen, K., Dong, Y ., and Wang, Y . Flashdecoding++: Faster large language model inference on gpus.arXiv preprint arXiv:2311.01282,

    Pith/arXiv arXiv

  3. [6]

    Hunyuan- video: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603,

    Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Wu, B., Zhang, J., et al. Hunyuan- video: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603,

    Pith/arXiv arXiv

  4. [7]

    Diffwave: A versatile diffusion model for audio synthesis

    Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro, B. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761,

    Pith/arXiv arXiv 2009

  5. [9]

    dkv-cache: The cache for diffusion language models.arXiv preprint arXiv:2505.15781,

    Ma, X., Yu, R., Fang, G., and Wang, X. dkv-cache: The cache for diffusion language models.arXiv preprint arXiv:2505.15781,

    Pith/arXiv arXiv

  6. [10]

    F., Arora, S., Singhal, A., Fu, D

    Spector, B. F., Arora, S., Singhal, A., Fu, D. Y ., and R ´e, C. Thunderkittens: Simple, fast, and adorable ai kernels. arXiv preprint arXiv:2410.20399,

    Pith/arXiv arXiv

  7. [11]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314,

    Wan, T., Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.-W., Chen, D., Yu, F., Zhao, H., Yang, J., et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314,

    Pith/arXiv arXiv

  8. [13]

    Sparse videogen: Acceler- ating video diffusion transformers with spatial-temporal sparsity.arXiv preprint arXiv:2502.01776,

    Xi, H., Yang, S., Zhao, Y ., Xu, C., Li, M., Li, X., Lin, Y ., Cai, H., Zhang, J., Li, D., et al. Sparse videogen: Acceler- ating video diffusion transformers with spatial-temporal sparsity.arXiv preprint arXiv:2502.01776,

    Pith/arXiv arXiv

  9. [14]

    Xattention: Block sparse attention with antidiagonal scoring.arXiv preprint arXiv:2503.16428,

    Xu, R., Xiao, G., Huang, H., Guo, J., and Han, S. Xattention: Block sparse attention with antidiagonal scoring.arXiv preprint arXiv:2503.16428,

    Pith/arXiv arXiv

  10. [15]

    Sparse videogen2: Accel- erate video generation with sparse attention via semantic- aware permutation.arXiv preprint arXiv:2505.18875,

    Yang, S., Xi, H., Zhao, Y ., Li, M., Zhang, J., Cai, H., Lin, Y ., Li, X., Xu, C., Peng, K., et al. Sparse videogen2: Accel- erate video generation with sparse attention via semantic- aware permutation.arXiv preprint arXiv:2505.18875,

    Pith/arXiv arXiv

  11. [16]

    Flash- infer: Efficient and customizable attention engine for llm inference serving.arXiv preprint arXiv:2501.01005,

    Ye, Z., Chen, L., Lai, R., Lin, W., Zhang, Y ., Wang, S., Chen, T., Kasikci, B., Grover, V ., Krishnamurthy, A., et al. Flash- infer: Efficient and customizable attention engine for llm inference serving.arXiv preprint arXiv:2501.01005,

    Pith/arXiv arXiv

  12. [17]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention.arXiv preprint arXiv:2502.11089,

    Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z., Xie, Z., Wei, Y ., Wang, L., Xiao, Z., et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention.arXiv preprint arXiv:2502.11089,

    Pith/arXiv arXiv

  13. [18]

    Zhang, J., Huang, H., Zhang, P., Wei, J., Zhu, J., and Chen, J. Sageattention2: Efficient attention with thorough out- lier smoothing and per-thread int4 quantization.arXiv preprint arXiv:2411.10958, 2024a.14 Zhang, J., Wei, J., Huang, H., Zhang, P., Zhu, J., and Chen, J. Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration.arX...

    arXiv

  14. [2020]

    Radial atten- tion: o(nlogn) sparse attention with energy decay for long video generation.arXiv preprint arXiv:2506.19852,

    Li, X., Li, M., Cai, T., Xi, H., Yang, S., Lin, Y ., Zhang, L., Yang, S., Hu, J., Peng, K., et al. Radial atten- tion: o(nlogn) sparse attention with energy decay for long video generation.arXiv preprint arXiv:2506.19852,

    arXiv

  15. [2022]

    Flex attention: A programming model for gen- erating optimized attention kernels.arXiv preprint arXiv:2412.05496,

    Dong, J., Feng, B., Guessous, D., Liang, Y ., and He, H. Flex attention: A programming model for gen- erating optimized attention kernels.arXiv preprint arXiv:2412.05496,

    Pith/arXiv arXiv

  16. [2023]

    S., Seo, J.-s., Zhang, Z., and Gupta, U

    Hu, Z., Meng, J., Akhauri, Y ., Abdelfattah, M. S., Seo, J.-s., Zhang, Z., and Gupta, U. Accelerating diffusion language model inference via efficient kv caching and guided diffusion.arXiv preprint arXiv:2505.21467,

    arXiv

  17. [2024]

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020.13 Esser, P., Kulal, S., Blattmann, A., Entezari, R., M¨uller, J., Saini, H., Levi, Y ., Lorenz, ...

    Pith/arXiv arXiv 2010

  18. [2025]

    Flashmask: Efficient and rich mask extension of flashattention.arXiv preprint arXiv:2410.01359,

    Wang, G., Zeng, J., Xiao, X., Wu, S., Yang, J., Zheng, L., Chen, Z., Bian, J., Yu, D., and Wang, H. Flashmask: Efficient and rich mask extension of flashattention.arXiv preprint arXiv:2410.01359,

    Pith/arXiv arXiv