Pith. sign in

REVIEW 3 major objections 5 minor 43 references

A hierarchy of pooled key blocks turns diffusion-transformer attention from quadratic to log-linear, preserving image quality while accelerating training and inference.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 15:28 UTC pith:O3GZJ236

load-bearing objection LLSA is a plausible hierarchical sparse-attention design with a real kernel and credible speedups, but missing seed variance and an untested routing assumption keep it from being airtight. the 3 major comments →

arxiv 2512.16615 v2 pith:O3GZJ236 submitted 2025-12-18 cs.CV

Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers

Yifan Zhou , Zeqi Xiao , Tianyi Wei , Shuai Yang , Xingang Pan This is my paper
classification cs.CV
keywords Sparse attentionDiffusion transformersHierarchical selectionTop-K attentionLog-linear complexityImage generationGPU kernelsLong sequences
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Diffusion transformers become prohibitively slow as token sequences grow because full self-attention costs O(N²) compute. Previous Top-K sparse attention reduced the attention stage but left a quadratic selection stage that dominates at long lengths. This paper identifies the single-level design as the root cause and proposes LLSA, which selects key blocks hierarchically over O(log N) pooled levels and enriches each query with coarse context. The total cost becomes O(N log N) with constant K, and the authors report a 28.27× faster attention inference and 6.09× faster training at 256×256 pixel tokens while maintaining generation quality. If the bound holds, it removes the main obstacle to training long-sequence diffusion transformers on a single GPU.

Core claim

LLSA replaces the single-level Top-K block selection used in prior sparse attention with a hierarchy: Q, K, V are mean-pooled across log(N) levels; the coarsest level computes full similarities and keeps the K best key blocks per query, and each finer level only scores the K·B descendants of blocks already selected, carrying indices downward. To preserve global context, each query's attention set appends one reweighted coarse key/value token from every level, with weights equal to the pool size B^l. The paper shows the selection stage costs O(NK) via a convergent geometric series and the attention stage O(NK log N), giving total O(N log N) for constant K; an efficient sparse-index transpose

What carries the argument

Hierarchical Top-K selection over mean-pooled levels, combined with Hierarchical KV Enrichment. Compression builds L=⌊log_B N−1⌋ levels; the coarsest level does full similarity scoring, and each finer level scores only the K·B candidates inherited from the coarse Top-K blocks, making selection O(NK) by a geometric sum. KV Enrichment appends one coarse token per level to every query's key/value set, with weights W(l)=B^l, so the effective attention set is O(K log N) per query and global context is retained without raising K. The supporting mechanism is a sparse-index transpose kernel (CSR-to-CSC scan) that gives the backward pass query indices per key without constructing a dense N×N mask.

Load-bearing premise

The load-bearing premise is that a query's important fine-scale keys are always descendants of its coarse-scale top-K blocks, so narrowing the search at each level never discards a key that would have been chosen at the full fine resolution.

What would settle it

Run LLSA's hierarchical selection on a batch of real queries and compare the resulting fine-level top-K sets against the exact global per-query top-K: if a substantial fraction of true top-K blocks fall outside the coarse-selected ancestors, the hierarchy is discarding information; a downstream test would train on data with many small high-contrast details and check whether FID diverges from full attention as sequence length grows.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the O(N log N) bound holds in practice, diffusion transformers can train on 65,536-token pixel sequences on a single GPU, removing the need for patchification or a VAE encoder.
  • The paper reports that K=8 suffices to outperform prior Top-K methods using K=20–32, implying that hierarchical context can substitute for a larger sparsity budget.
  • Because the backward pass uses only sparse indices, training throughput stays nearly constant across sequence lengths, so per-token cost stops growing with N.
  • The 512×512 experiments show throughput scaling consistent with O(N log N), suggesting the approach extends to even longer sequences such as video tokens.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same coarse-to-fine routing could be ported to video or 3D data with little modification, since locality in those modalities is generally even stronger than in 2D images – a testable extension the paper does not run.
  • The KV reweighting rule (importance proportional to pool size) is a heuristic; a learned or adaptive weighting could close the remaining FID gap to full attention and could be tested independently.
  • The constant-K claim depends on the hierarchy being a sufficient search structure; on data with many isolated salient pixels, a single-level full search may be necessary, so a hybrid that re-checks a few global candidates could be more robust.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes LLSA, a hierarchical block-sparse attention mechanism for diffusion transformers. It compresses Q/K/V into multiple logarithmic levels, performs coarse-to-fine Top-K selection by restricting finer-level candidates to descendants of coarser-level selections, and enriches the final attention with weighted coarse KV tokens. The authors claim O(N log N) complexity with constant K and B, and present a sparse-index GPU implementation that avoids dense masks in both forward and backward passes. Experiments on pixel-space DiTs (FFHQ-128/256/512 and ImageNet via PixelFlow) report large training and inference speedups over full attention and prior Top-K sparse-attention methods, with FID comparable or better in most settings.

Significance. If the claims hold, this is a valuable step for long-sequence DiT training. The complexity derivation is explicit and the proposed sparse-index transpose for the backward pass is a practical contribution that removes a quadratic dense-mask bottleneck. The experimental evaluation is broad, includes a realistic ImageNet-256 benchmark, and the code is released. The main risks are empirical: the central quality-maintenance claim rests on single-run FID differences that are often smaller than typical run-to-run noise, and the hierarchical routing assumption is never directly tested. These issues are addressable and do not invalidate the algorithmic idea.

major comments (3)
  1. [Algorithm 1 (Sec. 4.1)] Algorithm 1 is not self-contained and cannot be executed as written. In line 5, I^(L) is initialized as the vector [1,...,N/B^L], but it is used in lines 9-11 as if it had a row per query block (T^(L)=N/B^(L+1)). The prose says the full similarity S^(L)=Q^(L)K^(L)^T is computed on the coarsest level, which is not reflected in the pseudocode. Moreover, the mapping from a query block at level l+1 to its B child query blocks at level l is never defined, so I^(l) cannot be derived from I^(l+1) as written. Please provide a corrected, complete pseudocode and state explicitly how parent selections are propagated to finer query blocks.
  2. [Sec. 4.1, Alg. 1; Sec. 5.2, Table 1c] The hierarchical routing assumption is untested. At each level, selection is restricted to descendants of the coarse-level Top-K blocks; a fine block whose coarse ancestor ranks outside Top-K is never revisited. Mean pooling also averages away small-scale salient details. The paper never measures how often the true fine-level Top-K blocks are recovered, and end-to-end FID can be maintained by coarse KV enrichment even when fine selection is poor. Thus the 'small K is enough' claim in Table 1c is not yet established as a property of the hierarchical selection. Please add a direct routing-recall experiment (e.g., compare LLSA's selected fine blocks against exhaustive fine-level Top-K) or an ablation with identical enrichment and token budget that isolates the hierarchy's contribution.
  3. [Table 1a and Table 2] FID values are reported without variance or confidence intervals. In Table 1a, LLSA beats full attention by 0.54 FID (24.37 vs 24.91), while in Table 2 at 256x256 full attention beats LLSA by 0.52 FID (38.77 vs 39.29). These differences are within typical run-to-run variation for FID on such datasets, so the 'maintaining generation quality' claim is not yet supported by the reported numbers. Please report mean and standard deviation over at least three seeds, or a paired evaluation, for the central comparisons against full attention and the main baselines.
minor comments (5)
  1. [Sec. 4.2, Eq. (6)] The text says 'except the coarsest level' but the sum runs from l=0 to L-1, which includes L levels when the coarsest level is L. Please clarify the range or the indexing.
  2. [Sec. 4.1, KV Reweighting] The justification for W^(l)=B^l is heuristic. The relation between 'importance proportional to block size' and the softmax attention output is not formally stated. Please clarify the approximation or clearly label this as an empirically-motivated scaling.
  3. [Sec. 4.3] The claim that 'the probability of inter-program write conflicts is extremely low' is informal. Either provide a short analysis or remove the claim; the benchmark in Fig. 4 is the relevant evidence.
  4. [Throughout] Typos and formatting: 'Hierarchical KV Enchirment' in Sec. 5.2; 'ImagetNet' in Table 4; the phrase 'we apply an index-reorderingscheme' in Sec. 5.1. Please proofread.
  5. [Fig. 3] The caption does not state the Top-K values or effective token counts used in each panel, making the efficiency comparison harder to interpret. Please include these details.

Circularity Check

0 steps flagged

No significant circularity; the complexity and quality claims are derived from stated assumptions and external measurements, not from fitted inputs or self-citations.

full rationale

The central complexity claim (Sec. 4.2) is an arithmetic consequence of the algorithm definition: Eq. (6) sums per-level costs O(N/B^{l+1} * KB) to O(NK), and the sparse attention stage is O(NK log N) because each query attends to K fine tokens plus O(log N) coarse enriched tokens. This is derived, not fitted. The assumption that K stays constant is stated as a design choice, not obtained from the results. The empirical efficiency ratios (28.27x inference, 6.09x training) and FID numbers are direct measurements against full attention and baseline methods, so they are not predictions constructed to match inputs. KV Reweighting W=B^l is presented as a heuristic in Sec. 4.1 and is tested by ablation in Table 1a; choosing a weighting by ablation is parameter selection, not circularity. The hierarchical routing assumption (fine important blocks descend from coarse top-K blocks) is an untested empirical premise that could threaten recall quality, but it is an assumption inside the algorithm, not a circular derivation: the paper does not claim to prove it from the outputs. No load-bearing self-citation is present; references to prior hierarchical and multi-resolution attention work [38, 43] are external and contextual. Therefore no circular step can be exhibited under the required standard.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central claims rest on (i) hyperparameters K, B, L, L_e, and W selected by ablation on FFHQ-128, and (ii) structural assumptions about mean-pooling fidelity and hierarchical locality. The O(N log N) complexity result is a mathematical consequence of assuming K and B constant; the quality claim depends on the hierarchical routing assumption not being violated. No new physical entities are introduced.

free parameters (6)
  • Top-K parameter K = 8 (default)
    Number of selected key blocks per level; chosen from ablation Table 1c where K=8 outperforms K=32 baselines. This is a hyperparameter tuned on FFHQ-128 and then used across resolutions.
  • Block size B = 16 (default)
    Token block size for pooling and sparse attention; chosen from Table 1b where B=16 gives better quality than B=64 at similar token budgets.
  • Number of hierarchy levels L = L = floor(log_B N - 1); defaults L=2 or L=3 in experiments
    The maximum level is set by sequence length, but the actual L used is an experimental choice; Table 5b shows L=2 vs L=3 tradeoffs for FFHQ-512.
  • KV enrichment levels L_e = L_e = L (default)
    Ablation Table 5a shows more enrichment levels improve FID at throughput cost; default is set to L rather than derived from first principles.
  • KV reweighting W^(l) = W^(l) = B^l
    Hand-chosen scaling based on the nearest-upsample assumption; Table 1a shows it improves FID by about 1.8. It is a design choice, not a fitted constant, but it affects quality.
  • Noise rescale factor s = s = n/64 for images larger than 64x64
    Training trick to align SNR with 64x64 images; Table 5c selects it over timestep shift and logit-normal sampling. It affects convergence, not attention complexity.
axioms (5)
  • domain assumption Mean pooling of query/key blocks preserves sufficient routing information for top-K relevance.
    Used in Hierarchical Compression (Sec. 4.1). If pooling destroys the signal that distinguishes important keys from unimportant keys, hierarchical selection fails. Single-level top-K attention shares this assumption, but the hierarchy compounds it.
  • domain assumption Hierarchical locality: a query's fine-level top-K key blocks are descendants of its coarse-level top-K blocks.
    Load-bearing assumption in Hierarchical Top-K Selection (Alg. 1, Sec. 4.1). Each level only searches among KB candidates inherited from the coarser level, so any important key missed at a coarse level is unrecoverable.
  • ad hoc to paper Coarse tokens are approximately recoverable by nearest upsampling from their constituent fine tokens.
    Introduced to justify KV Reweighting W^(l)=B^l (Sec. 4.1). This assumption is not independently validated and is used to set the relative importance of coarse tokens.
  • domain assumption K and B remain constant as sequence length N grows.
    Used in Sec. 4.2 to turn O(N K log N) into O(N log N). If K must grow with N to maintain quality, the asymptotic claim weakens; the paper shows constant K works up to 65K tokens but does not prove it for longer sequences.
  • domain assumption After index reordering, spatially adjacent pixels in the flattened sequence are more relevant to each other.
    Required for 1D hierarchical pooling to reflect 2D structure (Sec. 5.1). The ablation in Table 5d shows it helps FID, but it is a data-dependent heuristic.

pith-pipeline@v1.3.0-alltime-deepseek · 14804 in / 17079 out tokens · 157390 ms · 2026-08-03T15:28:19.644395+00:00 · methodology

0 comments
read the original abstract

Diffusion Transformers (DiTs) set the state of the art in visual generation, yet their quadratic self-attention cost fundamentally limits scaling to long token sequences. Recent Top-K sparse attention approaches reduce the computation of DiTs by compressing tokens into block-wise representation and selecting a small set of relevant key blocks, but still suffer from (i) quadratic selection cost on compressed tokens and (ii) increasing K required to maintain model quality as sequences grow. We identify that their inefficiency is due to the single-level design, as a single coarse level is insufficient to represent the global structure. In this paper, we introduce Log-linear Sparse Attention (LLSA), a trainable sparse attention mechanism for extremely long token sequences that reduces both selection and attention costs from quadratic to log-linear complexity by utilizing a hierarchical structure. LLSA performs hierarchical Top-K selection, progressively adopting sparse Top-K selection with the indices found at the previous level, and introduces a Hierarchical KV Enrichment mechanism that preserves global context while using fewer tokens of different granularity during attention computation. To support efficient training, we develop a high-performance GPU implementation that uses only sparse indices for both the forward and backward passes, eliminating the need for dense attention masks. We evaluate LLSA on high-resolution pixel-space image generation without using patchification and VAE encoding. LLSA accelerates attention inference by 28.27x and DiT training by 6.09x on 256x256 pixel token sequences, while maintaining generation quality. The results demonstrate that LLSA offers a promising direction for training long-sequence DiTs efficiently. Code is available at: https://github.com/SingleZombie/LLSA

Figures

Figures reproduced from arXiv: 2512.16615 by Shuai Yang, Tianyi Wei, Xingang Pan, Yifan Zhou, Zeqi Xiao.

Figure 1
Figure 1. Figure 1: Comparison between a general Top-K sparse attention and our Log-linear Sparse Attention (LLSA). In the example, we use a token sequence of length N = 8, block size B = 2, Top-K parameter K = 1. To reduce the complexity of the selection stage from O(N 2 ) to O(N), we extend single-level selection to O(log N) levels. To achieve this, we compute the Top-K of the full sequence on the coarsest level and recursi… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of index reordering. The default raster in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Acceleration ratio of different attention methods compared to PyTorch Attention (FlashAttention2). We evaluate training and [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The throughput of sparse key-value backward. Experi [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The FID curves of different training strategies. Com [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The FID and Inception Score curves of the first [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The qualitative results of pixel space DiT-S using LLSA [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The qualitative comparison of SLA, VSA, and LLSA [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 14 linked inside Pith

  1. [1]

    Sparse-vdit: Unleashing the power of sparse attention to accelerate video diffusion transformers.arXiv preprint arXiv:2506.03065, 2025

    Pengtao Chen, Xianfang Zeng, Maosen Zhao, Peng Ye, Mingzhu Shen, Wei Cheng, Gang Yu, and Tao Chen. Sparse-vdit: Unleashing the power of sparse attention to accelerate video diffusion transformers.arXiv preprint arXiv:2506.03065, 2025. 2

    Pith/arXiv arXiv 2025

  2. [2]

    Pixelflow: Pixel-space generative models with flow.arXiv preprint arXiv:2504.07963, 2025

    Shoufa Chen, Chongjian Ge, Shilong Zhang, Peize Sun, and Ping Luo. Pixelflow: Pixel-space generative models with flow.arXiv preprint arXiv:2504.07963, 2025. 2, 3, 6, 7, 10

    Pith/arXiv arXiv 2025

  3. [3]

    On the importance of noise scheduling for diffu- sion models.arXiv preprint arXiv:2301.10972, 2023

    Ting Chen. On the importance of noise scheduling for diffu- sion models.arXiv preprint arXiv:2301.10972, 2023. 6

    Pith/arXiv arXiv 2023

  4. [4]

    Scalable high-resolution pixel-space image syn- thesis with hourglass diffusion transformers

    Katherine Crowson, Stefan Andreas Baumann, Alex Birch, Tanishq Mathew Abraham, Daniel Z Kaplan, and Enrico Shippole. Scalable high-resolution pixel-space image syn- thesis with hourglass diffusion transformers. InForty-first International Conference on Machine Learning, 2024. 3, 6

    2024

  5. [5]

    FlashAttention-2: Faster attention with better par- allelism and work partitioning

    Tri Dao. FlashAttention-2: Faster attention with better par- allelism and work partitioning. InInternational Conference on Learning Representations (ICLR), 2024. 3, 5

    2024

  6. [6]

    Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in neural information processing systems, 35:16344–16359, 2022

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christo- pher R ´e. Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in neural information processing systems, 35:16344–16359, 2022. 2, 3, 4

    2022

  7. [7]

    Scaling vision transformers to 22 billion pa- rameters

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdul- mohsin, et al. Scaling vision transformers to 22 billion pa- rameters. InInternational conference on machine learning, pages 7480–7512. PMLR, 2023. 6

    2023

  8. [8]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 2, 9

    2009

  9. [9]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. InForty-first International Conference on Machine Learn- ing, 2024. 6, 9

    2024

  10. [10]

    Log-linear attention.arXiv preprint arXiv:2506.04761, 2025

    Han Guo, Songlin Yang, Tarushii Goel, Eric P Xing, Tri Dao, and Yoon Kim. Log-linear attention.arXiv preprint arXiv:2506.04761, 2025. 3

    arXiv 2025

  11. [11]

    Two fast algorithms for sparse matri- ces: Multiplication and permuted transposition.ACM Trans- actions on Mathematical Software (TOMS), 4(3):250–269,

    Fred G Gustavson. Two fast algorithms for sparse matri- ces: Multiplication and permuted transposition.ACM Trans- actions on Mathematical Software (TOMS), 4(3):250–269,

  12. [12]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium.Advances in neural information processing systems, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium.Advances in neural information processing systems, 30, 2017. 6, 10

    2017

  13. [13]

    sim- ple diffusion: End-to-end diffusion for high resolution im- ages

    Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. sim- ple diffusion: End-to-end diffusion for high resolution im- ages. InInternational Conference on Machine Learning, pages 13213–13232. PMLR, 2023. 6

    2023

  14. [14]

    Minference 1.0: Accel- erating pre-filling for long-context llms via dynamic sparse attention.Advances in Neural Information Processing Sys- tems, 37:52481–52515, 2024

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, 10 Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accel- erating pre-filling for long-context llms via dynamic sparse attention.Advances in Neural Information Processing Sys- tems, 37:52481–52515, 2024. 2

    2024

  15. [15]

    Fast mul- tipole attention: A divide-and-conquer attention mechanism for long sequences.arXiv preprint arXiv:2310.11960, 2023

    Yanming Kang, Giang Tran, and Hans De Sterck. Fast mul- tipole attention: A divide-and-conquer attention mechanism for long sequences.arXiv preprint arXiv:2310.11960, 2023. 2

    arXiv 2023

  16. [16]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. InProceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 4401–4410, 2019. 6, 9

    2019

  17. [17]

    Reformer: The efficient transformer.arXiv preprint arXiv:2001.04451, 2020

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.arXiv preprint arXiv:2001.04451, 2020. 3

    Pith/arXiv arXiv 2001

  18. [18]

    Flux.https://github.com/ black-forest-labs/flux, 2023

    Black Forest Labs. Flux.https://github.com/ black-forest-labs/flux, 2023. 1

    2023

  19. [19]

    Radial attention: O (nlog n) sparse at- tention with energy decay for long video generation.arXiv preprint arXiv:2506.19852, 2025

    Xingyang Li, Muyang Li, Tianle Cai, Haocheng Xi, Shuo Yang, Yujun Lin, Lvmin Zhang, Songlin Yang, Jinbo Hu, Kelly Peng, et al. Radial attention: O (nlog n) sparse at- tention with energy decay for long video generation.arXiv preprint arXiv:2506.19852, 2025. 3

    arXiv 2025

  20. [20]

    Flow matching for generative mod- eling.arXiv preprint arXiv:2210.02747, 2022

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling.arXiv preprint arXiv:2210.02747, 2022. 6

    Pith/arXiv arXiv 2022

  21. [21]

    Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022. 6

    Pith/arXiv arXiv 2022

  22. [22]

    Moba: Mixture of block attention for long- context llms.arXiv preprint arXiv:2502.13189, 2025

    Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. Moba: Mixture of block attention for long- context llms.arXiv preprint arXiv:2502.13189, 2025. 2

    Pith/arXiv arXiv 2025

  23. [23]

    Sit: Explor- ing flow and diffusion-based generative models with scalable interpolant transformers.arXiv preprint arXiv:2401.08740,

    Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Explor- ing flow and diffusion-based generative models with scalable interpolant transformers.arXiv preprint arXiv:2401.08740,

    Pith/arXiv arXiv

  24. [24]

    Merge-based parallel sparse matrix-vector multiplication

    Duane Merrill and Michael Garland. Merge-based parallel sparse matrix-vector multiplication. InSC’16: Proceedings of the International Conference for High Performance Com- puting, Networking, Storage and Analysis, pages 678–689. IEEE, 2016. 5

    2016

  25. [25]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  26. [26]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 3

    2022

  27. [27]

    Improved techniques for training gans.Advances in neural information processing systems, 29, 2016

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans.Advances in neural information processing systems, 29, 2016. 10

    2016

  28. [28]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063,

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063,

  29. [29]

    Tri- ton: an intermediate language and compiler for tiled neu- ral network computations

    Philippe Tillet, Hsiang-Tsung Kung, and David Cox. Tri- ton: an intermediate language and compiler for tiled neu- ral network computations. InProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10–19, 2019. 7

    2019

  30. [30]

    Attention is all you need.Advances in Neural Information Processing Systems, 2017

    A Vaswani. Attention is all you need.Advances in Neural Information Processing Systems, 2017. 1

    2017

  31. [31]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianx- iao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025. 1

    Pith/arXiv arXiv 2025

  32. [32]

    Pixnerd: Pixel neural field diffusion

    Shuai Wang, Ziteng Gao, Chenhui Zhu, Weilin Huang, and Limin Wang. Pixnerd: Pixel neural field diffusion. 2025. 3

    2025

  33. [33]

    Vmoba: Mixture- of-block attention for video diffusion models.arXiv preprint arXiv:2506.23858, 2025

    Jianzong Wu, Liang Hou, Haotian Yang, Xin Tao, Ye Tian, Pengfei Wan, Di Zhang, and Yunhai Tong. Vmoba: Mixture- of-block attention for video diffusion models.arXiv preprint arXiv:2506.23858, 2025. 2, 7

    Pith/arXiv arXiv 2025

  34. [34]

    Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity.arXiv preprint arXiv:2502.01776, 2025

    Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, et al. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity.arXiv preprint arXiv:2502.01776, 2025. 2

    Pith/arXiv arXiv 2025

  35. [35]

    Training-free and adaptive sparse attention for efficient long video generation.arXiv preprint arXiv:2502.21079, 2025

    Yifei Xia, Suhan Ling, Fangcheng Fu, Yujie Wang, Huixia Li, Xuefeng Xiao, and Bin Cui. Training-free and adaptive sparse attention for efficient long video generation.arXiv preprint arXiv:2502.21079, 2025. 2

    Pith/arXiv arXiv 2025

  36. [36]

    Sparse videogen2: Accelerate video generation with sparse attention via semantic-aware permutation.arXiv preprint arXiv:2505.18875, 2025

    Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Jintao Zhang, Han Cai, Yujun Lin, Xiuyu Li, Chenfeng Xu, Kelly Peng, et al. Sparse videogen2: Accelerate video generation with sparse attention via semantic-aware permutation.arXiv preprint arXiv:2505.18875, 2025. 2

    Pith/arXiv arXiv 2025

  37. [37]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Proceedings of the 63rd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 23078–23097, 2025. 2

    2025

  38. [38]

    Multi resolution analysis (mra) for approx- imate self-attention

    Zhanpeng Zeng, Sourav Pal, Jeffery Kline, Glenn M Fung, and Vikas Singh. Multi resolution analysis (mra) for approx- imate self-attention. InInternational conference on machine learning, pages 25955–25972. PMLR, 2022. 2, 3

    2022

  39. [39]

    Spargeattention: Accurate and training-free sparse attention accelerating any model in- ference

    Jintao Zhang, Chendong Xiang, Haofeng Huang, Haocheng Xi, Jun Zhu, Jianfei Chen, et al. Spargeattention: Accurate and training-free sparse attention accelerating any model in- ference. InForty-second International Conference on Ma- chine Learning. 2, 6, 7

  40. [40]

    Gonzalez, Jun Zhu, and Jianfei Chen

    Jintao Zhang, Haoxu Wang, Kai Jiang, Shuo Yang, Kai- wen Zheng, Haocheng Xi, Ziteng Wang, Hongzhou Zhu, Min Zhao, Ion Stoica, Joseph E. Gonzalez, Jun Zhu, and Jianfei Chen. Sla: Beyond sparsity in diffusion transform- ers via fine-tunable sparse-linear attention.arXiv preprint arXiv:2509.24006, 2025. 2, 5, 7, 8, 9

    arXiv 2025

  41. [41]

    11 Faster video diffusion with trainable sparse attention.arXiv e-prints, pages arXiv–2505, 2025

    Peiyuan Zhang, Haofeng Huang, Yongqi Chen, Will Lin, Zhengzhong Liu, Ion Stoica, Eric P Xing, and Hao Zhang. 11 Faster video diffusion with trainable sparse attention.arXiv e-prints, pages arXiv–2505, 2025. 2, 5, 7, 9

    2025

  42. [42]

    Training-free efficient video generation via dynamic token carving.arXiv preprint arXiv:2505.16864, 2025

    Yuechen Zhang, Jinbo Xing, Bin Xia, Shaoteng Liu, Bo- hao Peng, Xin Tao, Pengfei Wan, Eric Lo, and Jiaya Jia. Training-free efficient video generation via dynamic token carving.arXiv preprint arXiv:2505.16864, 2025. 6

    arXiv 2025

  43. [43]

    H-transformer-1d: Fast one-dimensional hierarchical attention for sequences.arXiv preprint arXiv:2107.11906, 2021

    Zhenhai Zhu and Radu Soricut. H-transformer-1d: Fast one-dimensional hierarchical attention for sequences.arXiv preprint arXiv:2107.11906, 2021. 2 12 Figure 7. The qualitative results of pixel space DiT-S using LLSA trained on FFHQ-128, FFHQ-256, and FFHQ-512. For FFHQ- 512, the model is only trained for two epochs. We believe that better quality can be o...

    Pith/arXiv arXiv 2021