Pith. sign in

REVIEW 3 major objections 5 minor 79 references

Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Chipmunk claims that recomputing only the 5-25% of attention and MLP activations that change fastest each step — caching the rest — gives up to 2.16x speedup alone and 3.72x stacked with step caching, training-free.

desk verdict Solid systems paper with a genuine column-level caching idea, but the motivating R2 analysis does not measure the renormalized sparse softmax the kernels actually run, and the missing code makes the headline numbers unverifiable. read the letter →

arxiv 2506.03275 v1 pith:CJ4ETNKV submitted 2025-06-03 cs.CV cs.AI

Austin Silveria , Soham V. Govande , Daniel Y. Fu This is my paper
classification cs.CVcs.AI
keywords diffusiontransformerstraining-freeinferenceaccelerationdynamicsparsitycolumn-sparsekernelsactivationcachingcross-stepredundancytext-to-videogenerationtext-to-image
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to prove that diffusion transformer inference is doing redundant work: across the 50 denoising steps typical of high-quality video and image generation, only 5-25% of the intermediate values in attention and MLP layers account for 70-90% of the change in activations from one step to the next. On that evidence it builds Chipmunk, a training-free method that caches the slowly changing activations and recomputes only a dynamically selected sparse subset — the 'delta' — at each step, using the shared two-matrix-multiply structure of attention and MLPs to decide which vectors matter. To make the sparse recomputation faster than running the dense model, Chipmunk reorders tokens into 3D voxel chunks so the sparsity is column-structured, and uses custom kernels that gather sparse columns into dense shared-memory tiles to keep GPU tensor cores saturated. The claimed payoff is up to 2.16x end-to-end speedup on HunyuanVideo and 1.41x on FLUX.1-dev with essentially unchanged generation quality, rising to 3.72x and 2.25x when stacked on step caching. If the concentration finding holds, large generative diffusion transformers can be accelerated by 2-4x with no retraining and no change to the model weights, with the paper itself scoping the claim to large compute-bound models where caching headroom exists.

What carries the argument

The load-bearing device is the shared computational form of attention and MLP layers — $\mathrm{softmax}(QK^\top)V$ for attention and $\mathrm{GELU}(XW_1^\top)W_2$ for the MLP — in which a nonlinearity converts a first matrix product into scalar coefficients for a linear combination of vectors. Because each scalar in the intermediate activation matrix scales exactly one output vector (rows of $V$ in attention, rows of $W_2$ in the MLP), choosing which scalars to recompute one-to-one determines which vectors are refreshed from the cache, and the paper shows the DiT latent path is a sum of such individually scaled vectors. Chipmunk makes this selection hardware-friendly by requiring column-structured sparsity at granularity $[C,1]$ — contiguous chunks of $C=192$ tokens share the same selected columns — by reordering tokens so each chunk is a video voxel or image patch, and by caching the top-$k$ index mask at a dense step for reuse over several sparse steps. A second set of mechanisms makes the approximation cheap: attention reuses softmax normalization constants from the previous step, the MLP selects neurons from chunk-mean pre-activation deltas, and fused kernels overlap pattern identification and cache writeback with the main GEMMs.

What would settle it

Generate a video with an abrupt content change — a hard cut, a sudden camera pan, or a new object entering the frame — between a dense step and the following sparse steps, and measure the $R^2$ of Chipmunk's sparse-delta approximation against the true activation change; if the cached mask's explained variance collapses well below the reported 70-90%, the mask-staleness premise fails. A second check is kernel-level: run the column-sparse kernel at 93% sparsity on a GPU with roughly half the H100's shared memory and compare the 9.3x speedup against FlashAttention-3, since the paper itself reports that at very high sparsity fine detail such as hands and pencils begins to warp.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that cross-step change in DiT activations is highly concentrated: measuring the $R^2$ between the true activation change and the change captured by recomputing only a top-$k$ subset, the paper reports that 5% of attention values explain 92.4% of the change on HunyuanVideo, 25% of attention values explain 90.7% on FLUX.1-dev, and 15-25% of MLP activations explain roughly 70% of the change. This concentration means a layer's output at the current step can be approximated by the cached output from the previous step plus a sparse correction that recomputes only the fastest-changing columns of $K^\top$ or $W_1$ and the corresponding rows of $V$ or $W_2$, exactly the structure of Equations 1 and 2. Chipmunk operationalizes that approximation with column-sparse masks of granularity $[C,1]$ ($C=192$ on H100), a voxel reordering of tokens so each contiguous chunk is a spatiotemporal cube, and custom kernels that pack the selected sparse columns into dense shared-memory tiles; it then overlaps mask selection and cache updates with other computation. The end-to-end claim is that this achieves 2.16x speedup on HunyuanVideo and 1.41x on FLUX.1-dev with minimal impact on VBench, ImageReward, and CLIP scores, and that stacking with step caching reaches 3.72x, 2.67x, and 2.25x on HunyuanVideo, WAN2.1, and FLUX.1-dev with minimal quality impact.

Load-bearing premise

The central assumption is that activations change slowly and smoothly enough that a sparsity mask picked at the last dense step still points at the right attention and MLP interactions several steps later, with a second premise being that packing the selected sparse columns into dense shared-memory tiles keeps tensor cores saturated.

Editorial extensions

If this is right

  • If the concentration finding is right, any large DiT can be accelerated by roughly 2x on its own, and by about 2.25-3.72x when stacked with step caching, without retraining, by spending compute only on the 5-25% of interactions that actually change between steps.
  • Chipmunk's per-vector granularity is finer than existing step-, layer-, and token-level caching, so it composes with those methods rather than replacing them; the paper demonstrates stacking with step caching and with sliding-tile attention on WAN2.1.
  • The method transfers across inference regimes: it holds on attention-bound video models with 76k-118k token sequences (HunyuanVideo, WAN2.1) and on an MLP-bound image model with a roughly 4.5k sequence (FLUX.1-dev), so the finding is not specific to one architecture.
  • Because Chipmunk's sparse kernels accept any static index set, existing static-sparsity schemes such as sliding-window attention can run inside the same kernels, making column-sparse deltas a general substrate for DiT attention acceleration.
  • At the kernel level, the 9.3x speedup over FlashAttention-3 at 93% sparsity implies that column sparsity can be nearly as fast as block sparsity while producing about half the approximation error, which shifts the cost-quality trade-off for sparse attention generally.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The concentration-of-change finding is stated for diffusion denoising, but the underlying mechanism — residual-stream paths built from individually scaled attention and MLP vectors — also appears in autoregressive decoding and recurrent-depth transformers; if the same 5-25% concentration holds there, Chipmunk's sparse-delta strategy could transfer to LLM and autoregressive video inference without
  • The paper's own failure modes (warping of hands and fine motion at very high sparsity, blurry backgrounds) suggest the mask-staleness assumption, not kernel speed, is the practical ceiling; an untested extension would be refreshing the mask more often or using motion cues to predict which voxels will change, which should recover quality on videos with cuts or fast camera motion.
  • The attention approximation reuses softmax normalization constants from the previous step, and the paper does not characterize how the error of that stale constant grows with the number of sparse steps since the last dense step; measuring that growth would give a principled rule for choosing the dense-step interval.
  • The speedups are measured on H100 hardware with $C=192$ column chunks; on GPUs with less shared memory or different tensor-core shapes the packing efficiency will differ, so a testable prediction is that the end-to-end speedup tracks the ability to keep tiles of at least $64 \times 64$ resident in shared memory.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces Chipmunk, a training-free method to accelerate DiT inference by caching slowly changing intermediate activations and recomputing, at each sparse step, only a dynamically selected subset of column-chunks in attention and MLP computations. The authors first report a concentration finding (5-25% of intermediate values explain 70-90% of cross-step activation change), then design voxel-based token reordering and custom column-sparse CUDA kernels with fused sparsity-pattern computation and cache updates. They report kernel speedups (9.3x at 93% sparsity vs FlashAttention-3), end-to-end speedups of 2.16x on HunyuanVideo and 1.41x on FLUX.1-dev with no quality loss, and 3.72x/2.67x/2.25x when stacked with step caching on HunyuanVideo, WAN2.1, and FLUX.1-dev.

Significance. If the results hold, Chipmunk would be a valuable contribution: it achieves finer-grained caching than prior per-step/per-layer/per-token methods, shows a hardware-aware way to make dynamic column sparsity tensor-core efficient, and demonstrates large wall-clock speedups on state-of-the-art video and image DiTs. The kernel engineering (fused column-sum, persistent-grid/warp-specialized MLP delta, bitpacked masks, CPU offloading) is concrete and the paper includes reference algorithms. However, the central empirical justification, the concentration claim, rests on an approximation (masked full softmax) that differs from the deployed kernel (restricted softmax), so the motivating numbers do not currently validate the method as implemented.

major comments (3)
  1. [§3.3.1, Eq. (1), Alg. 3, Table 1] The algorithm actually deployed for attention computes the sparse delta with a softmax normalized only over the selected key subset (Alg. 3: p = online_softmax(qk) on gathered qk), whereas Eq. (1) defines the approximation as a full softmax whose probabilities are then masked. These are different functions: the restricted softmax redistributes probability mass among the selected keys, so the recomputed contribution is not the masked full-softmax term of Eq. (1). The R2 reported in Table 1 measures the masked full-softmax oracle, so it does not characterize the deployed approximation; the gap is unquantified and directly affects the central claim that '5-25% of the values explain 70-90% of the change' and the resulting quality/speed tradeoff. Please either modify the algorithm to use the full softmax normalization constants (e.g., stale denominator) so that it matches Eq. (1), or re-measure the R2 of the actual kernel.
  2. [§3.3.1 and Appendix B.2] The sparse mask used in deployed attention is not the oracle mask of Table 1. Column sums in Alg. 4 are approximated using the previous step's softmax normalization constants (Appendix B.2), and the resulting indices are cached at the dense step and reused for multiple sparse steps (Alg. 2). Table 1's R2 is computed with a per-value mask chosen from the current step's true activations, so it does not account for the staleness of the mask or the coarseness of the [C,1] column chunks. Please report an R2 or error metric for the end-to-end algorithm (including stale selection and chunking), and ablate the effect of mask staleness, since the reported failure modes (Fig. 10) suggest sensitivity to this.
  3. [§4.1, Tables 4-6] The experimental protocol reports single point estimates without variance: sparsity levels are tuned on a 100-generation warm-up, the step schedule is hand-chosen, and no error bars or repeated runs are shown. The speedup measurement for the STA baseline is obtained by replacing the attention kernel runtime rather than running the full end-to-end method (Appendix D.1). This makes it difficult to assess whether the claimed speedups and quality are robust or whether the comparison is fair. Please provide multiple seeds/prompts with standard errors, and describe the warm-up selection criterion and the exact baseline measurement procedures.
minor comments (5)
  1. [Alg. 3] The function online_softmax is not defined; please state whether it uses only the gathered qk tile or incorporates the previous step's softmax constants, since this is essential to clarifying the major comment above.
  2. [Appendix A, Eq. (11)-(13)] The notation mixes the timestep index t and layer indices; clarify the aggregation order and define st, We, Wu, and the LNt arguments precisely.
  3. [Appendix B.1] The text mentions a codebase with Triton kernels but no URL is provided; an anonymized link would help reproducibility.
  4. [Table 2 and Fig. 3] The runtime percentages should state whether they are measured on the deployed model configurations or on synthetic shapes, and the exact sparsity values used should be listed.
  5. [§4.1] The phrase '650 TFLOP FlashAttention-3 baseline' is ambiguous; specify whether this is the kernel peak FLOPS or a throughput measurement.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the concentration finding and speedups are empirical, externally benchmarked, and not constructed from the method's own outputs.

full rationale

Chipmunk's derivation chain is self-contained at the level of evidence. The motivating concentration claim (Table 1, "just 5-25% of the values in attention and MLP explain 70-90% of the change in activations across steps") is an R2 measurement of how well a sparse replacement of the largest cross-step activations reproduces the true activation delta; it is not an identity, since a uniform top-5% would explain only roughly 5% of the total squared change. The end-to-end quality claims are checked against external benchmarks (VBench, ImageReward, GenEval, CLIP) and the speedups are wall-clock measurements on H100, so neither reduces to a fitted parameter. The hyperparameter warm-up in Section 4.1 selects sparsity levels to hit a target fraction of explained change; this is calibration, not prediction. The self-citations (FlashAttention, ThunderKittens) are kernel/tool references and are not used as a load-bearing uniqueness or justification chain. The skeptical observation that Alg. 2-3's sparse attention renormalizes softmax over the selected key subset while Eq. 1 uses a masked full softmax identifies a genuine internal-consistency/validation gap: Table 1's R2 characterizes the Eq. 1 oracle, not the deployed kernel. That is a correctness/empirics concern, not a circularity, because the deployed approximation is not obtained by defining its output as its input; its quality is still measured against external benchmarks and wall-clock time. The Limitations section (Sec. 6) notes reduced benefit for small models and few-step models, which is an honest scope caveat and does not indicate circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

Chipmunk's central claim rests on empirical regularities measured on the same models it evaluates: (1) cross-step activation change concentrates in a small fraction of activations (Table 1), (2) stale masks track that concentration over several sparse steps, and (3) voxel-local tokens co-activate on shared columns so that column packing preserves tensor-core throughput. All three are model-specific measurements, so the claim's reach is bounded to these architectures and the fitted sparsity levels. There are no invented physical entities; the Appendix A 'vector decomposition' of the residual stream is an interpretive framework, not a new mechanism.

free parameters (6)
  • Attention sparsity level per model = 95% HunyuanVideo, 82% WAN2.1, 84% FLUX.1-dev
    Selected in a 100-generation warm-up to capture 95% of activation change (Sec. 4.1, D.1); modulates the speed-quality tradeoff in Tables 4-6.
  • MLP sparsity level = 70% for FLUX.1-dev; sparsity not applied to video model MLPs
    Warm-up selected; MLP is a small FLOP share in video models (Fig. 3, right).
  • Token chunk size C = 192
    Hand-chosen for H100 efficiency (Sec. 3.2.1: 'on H100 GPUs, we find C = 192 is efficient').
  • Voxel shapes = (4,6,8) for WAN2.1 STA stacking; others unspecified
    Voxel-dimension parameters of Alg. 1 are set by hand; only the WAN2.1 configuration is given numerically (D.1).
  • Step schedule = 1 dense per 10 sparse steps; W=30, n=4 when stacked with step caching
    Stated as a simple, untuned schedule (Sec. 4.1, D.1); it contributes roughly 1.8x of the stacked speedup.
  • Approximate top-k error/size = not specified
    Table 3 credits 'Approximate top-k' with a 3.42x speedup but gives no approximation parameters or error bounds.
assumptions (5)
  • domain assumption DiT intermediate activations change slowly and predictably across inference steps, so masks computed at a dense step or with previous-step softmax constants remain valid for several subsequent sparse steps.
    Core caching premise. Invoked in Alg. 2-3 (cached indices reused across sparse steps) and B.2 ('we reuse softmax normalization constants from the previous inference step'). The warping failure modes (Fig. 10) are consistent with this premise breaking under fast motion.
  • domain assumption Top-k selection by column-sum attention probabilities (attention) and chunk-mean pre-activation deltas (MLP) identifies the activations with the largest cross-step change.
    Selection proxies in Sec. 3.3.1-3.3.2. Table 1's R2 uses an oracle per-value mask; the deployed proxies are coarser and the gap is unquantified.
  • domain assumption Spatiotemporally local tokens (voxels) activate overlapping sets of keys/values and neurons, so a contiguous 192-token chunk can share one sparse column set.
    Justifies voxel reordering (Alg. 1) and the column-sparse hardware mapping (Fig. 1-2). Fig. 2 shows the benefit is attention-specific (nearly zero MLP effect).
  • domain assumption Tensor cores need dense SRAM tiles of at least 64x64 for peak throughput, so sparse compute must be packed into dense tiles.
    Hardware premise from Sec. 2.2 (citing [31]) that motivates the sparse-gather kernels; the claimed speedups inherit this precondition.
  • ad hoc to paper The DiT residual stream can be decomposed into a sum of individually scaled attention/MLP vectors, with modulation, layernorm scale-shift, and linear projections distributing over the sum.
    Appendix A (Eqs. 11-13). The authors concede layernorm's mean/variance are non-distributive and treat them as a two-phase read/write; this framework motivates the method but is not load-bearing for the empirical speedups.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas." pith.science (2026). https://pith.science/paper/CJ4ETNKV

@misc{pith2026250603275,
  author       = {Pith},
  title        = {Pith review of: Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CJ4ETNKV}},
  note         = {Machine review of arXiv:2506.03275}
}
read the original abstract

Diffusion Transformers (DiTs) have achieved state-of-the-art performance in high-quality image and video generation but incur substantial compute cost at inference. A common observation is that DiT latent noise vectors change slowly across inference steps, which suggests that the DiT compute may be redundant across steps. In this paper, we aim to speed up inference by reducing this redundancy, without additional training. We first study how activations change between steps in two state-of-the-art open-source DiTs. We find that just 5-25% of the values in attention and MLP explain 70-90% of the change in activations across steps. This finding motivates our approach, Chipmunk, which uses dynamic sparsity at inference time to recompute only the fastest-changing intermediate activations, while caching the rest. Dynamic sparsity introduces two systems challenges: (1) sparse attention and MLP operations tend to underutilize GPU tensor cores; and (2) computing dynamic sparsity patterns at runtime and caching activations both introduce overhead. To address these challenges, Chipmunk first uses a voxel-based reordering of input tokens to introduce column-wise sparsity. We implement column-sparse kernels utilizing efficient sparse gathers from global to shared GPU memory, achieving a 9.3x speedup at 93% sparsity compared to highly-optimized dense baselines. Second, Chipmunk overlaps the computation of sparsity patterns and cache updates with other parts of the computation (e.g., second layer of the MLP) to hide the extra latency. Chipmunk achieves up to 2.16x speedup on HunyuanVideo and 1.41x on FLUX.1-dev without compromising generation quality. Furthermore, we show that Chipmunk can be stacked on top of full step caching, achieving a 3.72x speedup on HunyuanVideo, a 2.67x speedup on WAN2.1, and a 2.25x speedup on FLUX.1-dev with minimal quality impact.

Figures

Figures reproduced from arXiv: 2506.03275 by the authors.

Figure 1
Figure 1. Left: Chipmunk uses intermediate activation sparsity to recompute only the fastest changing vectors in the output linear combinations of attention and MLPs. Middle: Column sparse kernels achieve low approximation error and hardware-efficiency by packing dense SRAM tiles for peak tensor core utilization. Right: Extra operations required to compute dynamic sparse deltas, such as sparsity pattern identification and cac… view at source ↗
Figure 2
Figure 2. Left: Chipmunk reorders tokens such that column sparse patterns route voxels (3D cubes of pixels) to the same set of activated keys/values (attention) or model weights (columns/rows of W1/W2 in MLPs). Right: We plot the unexplained variance in cross-step activation changes (1 - R2 ). Relative to dynamic block sparsity, dynamic column sparsity gives a 2x reduction in unexplained variance across both attention and MLP… view at source ↗
Figure 3
Figure 3. Left, Middle: Sparse kernel runtime scales linearly with sparsity. Column sparsity is competitive with block sparsity. Right: FLOP breakdown by model at Chipmunk hyperparameters. These findings motivate Chipmunk’s caching approach ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Qualitative comparisons across videos (left) and images (right). For videos, frames are [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: An individual-vector interpetation of the full HunyuanVideo DiT architecture. Attention [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Left: Both MLP and attention operations use a non-linearity to compute the scalar coefficients [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Images on 1280x768 FLUX.1-dev evaluated on different captions and prompts randomly [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]
Figure 8
Figure 8. Figure 8: Speed-quality tradeoff on 1280x768 images generated on FLUX.1-dev with 50 steps at [PITH_FULL_IMAGE:figures/full_fig_p028_8.png]
Figure 9
Figure 9. Figure 9: Ablation of adding deltas (to reuse attention interactions that are not recomputed) and 1% [PITH_FULL_IMAGE:figures/full_fig_p029_9.png]
Figure 10
Figure 10. Figure 10: Comparing the quality of WAN2.1 generation at 1% dynamic top-k attention interactions [PITH_FULL_IMAGE:figures/full_fig_p030_10.png]
Figure 11
Figure 11. Figure 11: Qualitative video generation comparison across different methods for HunyuanVideo [PITH_FULL_IMAGE:figures/full_fig_p031_11.png]
Figure 12
Figure 12. Figure 12: Qualitative video generation comparison across different methods for WAN2.1. [PITH_FULL_IMAGE:figures/full_fig_p032_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

79 extracted references · 32 canonical work pages

  1. [1]

    Arora, S

    S. Arora, S. Eyuboglu, M. Zhang, A. Timalsina, S. Alberti, D. Zinsley, J. Zou, A. Rudra, and C. Ré. Simple linear attention language models balance the recall-throughput tradeoff, 2025. URL https://arxiv.org/abs/2402.18668

    arXiv 2025

  2. [2]

    B. Chen, T. Dao, E. Winsor, Z. Song, A. Rudra, and C. Ré. Scatterbrain: Unifying sparse and low-rank attention approximation, 2021. URLhttps://arxiv.org/abs/2110.15343

    work page Pith review arXiv 2021

  3. [3]

    J. Chen, J. Yu, C. Ge, L. Yao, E. Xie, Y . Wu, Z. Wang, J. Kwok, P. Luo, H. Lu, and Z. Li. Pixart-α: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023. URL https://iclr.cc/media/iclr-2024/Slides/18231.pdf

    work page 2023

  4. [4]

    P. Chen, M. Shen, P. Ye, J. Cao, C. Tu, C.-S. Bouganis, Y . Zhao, and T. Chen. δ-dit: A training-free acceleration method tailored for diffusion transformers, 2024. URL https://arxiv.org/abs/2406.01125

    arXiv 2024

  5. [5]

    Y . Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia. Longlora: Efficient fine-tuning of long-context large language models, 2024. URLhttps://arxiv.org/abs/2309.12307

    arXiv 2024

  6. [6]

    Z. Chen, Z. Qu, L. Liu, Y . Ding, and Y . Xie. Efficient tensor core-based gpu kernels for structured sparsity under reduced precision. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’21, New York, NY , USA, 2021. Association for Computing Machinery. ISBN 9781450384421. doi: 10.1145/3458...

    arXiv 2021

  7. [7]

    Choromanski, V

    K. Choromanski, V . Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser, D. Belanger, L. Colwell, and A. Weller. Rethinking attention with performers, 2022. URL https://arxiv.org/abs/2009.14794

    arXiv 2022

  8. [8]

    T. Dao, D. Y . Fu, S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. URLhttps://arxiv.org/abs/2205.14135

    arXiv 2022

Show all 79 references
  1. [9]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://iclr.cc/virtual/2021/po...

    2021

  2. [10]

    Elhage, N

    N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y . Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, a...

  3. [11]

    Esser, S

    P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y . Levi, D. Lorenz, A. Sauer, F. Boesel, D. Podell, T. Dockhorn, Z. English, K. Lacey, A. Goodwin, Y . Marek, and R. Rombach. Scaling rectified flow transformers for high-resolution image synthesis, 2024. URL...

    2024

  4. [12]

    Fedus, B

    W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022. URLhttps://arxiv.org/abs/2101.03961

    2022 arXiv

  5. [13]

    Geiping, S

    J. Geiping, S. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. R. Bartoldson, B. Kailkhura, A. Bhatele, and T. Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05171, 2025

    2025 arXiv

  6. [14]

    Ghosh, H

    D. Ghosh, H. Hajishirzi, and L. Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment, 2023. URL https://arxiv.org/abs/2310.11513

    2023 arXiv

  7. [15]

    J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models, 2020. URL https://arxiv.org/abs/2006.11239

    2020 arXiv

  8. [16]

    W. Hong, M. Ding, W. Zheng, X. Liu, and J. Tang. Cogvideo: Large-scale pretraining for text- to-video generation via transformers, 2022. URLhttps://arxiv.org/abs/2205.15868

    2022 arXiv

  9. [17]

    Huang, Y

    Z. Huang, Y . He, J. Yu, F. Zhang, C. Si, Y . Jiang, Y . Zhang, T. Wu, Q. Jin, N. Chanpaisit, Y . Wang, X. Chen, L. Wang, D. Lin, Y . Qiao, and Z. Liu. Vbench: Comprehensive bench- mark suite for video generative models, 2023. URL https://openaccess.thecvf.com/ content/CVPR202...

    2023

  10. [18]

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T...

    2024 arXiv

  11. [19]

    Kahatapitiya, H

    K. Kahatapitiya, H. Liu, S. He, D. Liu, M. Jia, C. Zhang, M. S. Ryoo, and T. Xie. Adaptive caching for faster video generation with diffusion transformers, 2024. URL https://arxiv.org/abs/2411.02397

    2024 arXiv

  12. [20]

    Kitaev, Łukasz Kaiser, and A

    N. Kitaev, Łukasz Kaiser, and A. Levskaya. Reformer: The efficient transformer, 2020. URL https://arxiv.org/abs/2001.04451

    2020 arXiv

  13. [21]

    W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603 . URL https://ui.adsabs.harvard.edu/abs/ 2024arXiv241203603K/abstract

    arXiv

  14. [22]

    B. F. Labs. Flux. https://github.com/black-forest-labs/flux , 2024

    2024

  15. [23]

    S. Li, K. Osawa, and T. Hoefler. Efficient quantized sparse matrix operations on tensor cores. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, page 1–15. IEEE, Nov. 2022. doi: 10.1109/sc41404.2022.00042. URL http://dx.doi.org...

    2022 arXiv

  16. [24]

    Z. Li, C. You, S. Bhojanapalli, D. Li, A. S. Rawat, S. J. Reddi, K. Ye, F. Chern, F. Yu, R. Guo, et al. The lazy neuron phenomenon: On emergence of activation sparsity in transformers.arXiv preprint arXiv:2210.06313, 2022. 12

    2022 arXiv

  17. [25]

    T.-Y . Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár. Microsoft coco: Common objects in context, 2015. URL https://arxiv.org/abs/1405.0312

    2015 arXiv

  18. [26]

    F. Liu, S. Zhang, X. Wang, Y . Wei, H. Qiu, Y . Zhao, Y . Zhang, Q. Ye, and F. Wan. Timestep embed- ding tells: It’s time to cache for video diffusion model.arXiv preprint arXiv:2411.19108, 2024

    2024 arXiv

  19. [27]

    J. Liu, P. Ponnusamy, T. Cai, H. Guo, Y . Kim, and B. Athiwaratkun. Training-free activation sparsity in large language models.arXiv preprint arXiv:2408.14690, 2024

    2024 arXiv

  20. [28]

    X. Liu, C. Gong, and Q. Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow, 2022. URLhttps://iclr.cc/virtual/2023/oral/12626

    2022

  21. [29]

    Z. Liu, J. Wang, T. Dao, T. Zhou, B. Yuan, Z. Song, A. Shrivastava, C. Zhang, Y . Tian, C. Re, and B. Chen. Deja vu: Contextual sparsity for efficient llms at inference time, 2023. URL https://arxiv.org/abs/2310.17157

    2023 arXiv

  22. [30]

    C. Lu, Y . Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps, 2022. URL https://arxiv.org/abs/2206.00927

    2022 arXiv

  23. [31]

    W. Luo, R. Fan, Z. Li, D. Du, Q. Wang, and X. Chu. Benchmarking and dissecting the nvidia hop- per gpu architecture, 2024. URL https://ieeexplore.ieee.org/document/10579250

    2024

  24. [32]

    X. Ma, G. Fang, and X. Wang. Deepcache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15762–15772, 2024

    2024

  25. [33]

    Gpu performance background user’s guide, 2024

    NVIDIA. Gpu performance background user’s guide, 2024. URL https://docs.nvidia. com/deeplearning/performance/dl-performance-gpu-background/index.html

    2024

  26. [34]

    Peebles and S

    W. Peebles and S. Xie. Scalable diffusion models with transformers, 2023. URL https://openaccess.thecvf.com/content/ICCV2023/html/Peebles_Scalable_ Diffusion_Models_with_Transformers_ICCV_2023_paper.html

    2023

  27. [35]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https://arxiv.org/abs/2103.00020

    2021 arXiv

  28. [36]

    Salimans and J

    T. Salimans and J. Ho. Progressive distillation for fast sampling of diffusion models, 2022. URL https://arxiv.org/abs/2202.00512

    2022 arXiv

  29. [37]

    Magi-1: Autoregressive video generation at scale, 2025

    Sand-AI. Magi-1: Autoregressive video generation at scale, 2025. URL https: //static.magi.world/static/files/MAGI_1.pdf

    2025

  30. [38]

    J. Shah, G. Bikshandi, Y . Zhang, V . Thakkar, P. Ramani, and T. Dao. Flashattention- 3: Fast and accurate attention with asynchrony and low-precision, 2024. URL https://tridao.me/blog/2024/flash3/

    2024

  31. [39]

    Shazeer, A

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outra- geously large neural networks: The sparsely-gated mixture-of-experts layer, 2017. URL https://arxiv.org/abs/1701.06538

    2017 arXiv

  32. [40]

    Sohl-Dickstein, E

    J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli. Deep un- supervised learning using nonequilibrium thermodynamics, 2015. URL https: //proceedings.mlr.press/v37/sohl-dickstein15.html

    2015

  33. [41]

    J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models, 2022. URL https://arxiv.org/abs/2010.02502

    2022 arXiv

  34. [42]

    Y . Song, P. Dhariwal, M. Chen, and I. Sutskever. Consistency models, 2023. URL https://proceedings.mlr.press/v202/song23a.html

    2023

  35. [43]

    B. F. Spector, S. Arora, A. Singhal, D. Y . Fu, and C. Ré. Thunderkittens: Simple, fast, and adorable ai kernels, 2024. URLhttps://iclr.cc/virtual/2025/poster/31243. 13

    2024

  36. [44]

    X. Sun, J. Fang, A. Li, and J. Pan. Unveiling redundancy in diffusion transformers (dits): A systematic study, 2024. URLhttps://arxiv.org/abs/2411.13588

    2024 arXiv

  37. [45]

    X. Tan, Y . Chen, Y . Jiang, X. Chen, K. Yan, N. Duan, Y . Zhu, D. Jiang, and H. Xu. Dsv: Exploiting dynamic sparsity to accelerate large-scale video dit training, 2025. URL https://arxiv.org/abs/2502.07590

    2025

  38. [46]

    J. Tang, Y . Zhao, K. Zhu, G. Xiao, B. Kasikci, and S. Han. Quest: Query-aware sparsity for efficient long-context llm inference, 2024. URLhttps://arxiv.org/abs/2406.10774

    2024 arXiv

  39. [47]

    Tillet, H

    P. Tillet, H. T. Kung, and D. Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY , USA, 2019. ...

    2019

  40. [48]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, 2023. URLhttps://arxiv.org/abs/1706.03762

    2023 arXiv

  41. [49]

    S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma. Linformer: Self-attention with linear complexity, 2020. URL https://arxiv.org/abs/2006.04768

    2020 arXiv

  42. [50]

    Wimbauer, B

    F. Wimbauer, B. Wu, E. Schoenfeld, X. Dai, J. Hou, Z. He, A. Sanakoyeu, P. Zhang, S. Tsai, J. Kohler, C. Rupprecht, D. Cremers, P. Vajda, and J. Wang. Cache me if you can: Accelerating diffusion models through block caching, 2024. URLhttps://arxiv.org/abs/2312.03209

    2024 arXiv

  43. [51]

    H. Xi, S. Yang, Y . Zhao, C. Xu, M. Li, X. Li, Y . Lin, H. Cai, J. Zhang, D. Li, J. Chen, I. Stoica, K. Keutzer, and S. Han. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity, 2025. URLhttps://arxiv.org/abs/2502.01776

    2025 arXiv

  44. [52]

    J. Xu, X. Liu, Y . Wu, Y . Tong, Q. Li, M. Ding, J. Tang, and Y . Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Proces...

  45. [53]

    R. Xu, G. Xiao, H. Huang, J. Guo, and S. Han. Xattention: Block sparse attention with antidiagonal scoring, 2025. URL https://arxiv.org/abs/2503.16428

    2025 arXiv

  46. [54]

    J. Yao, W. Cheng, W. Liu, and X. Wang. Fasterdit: Towards faster diffusion transformers training without architecture modification, 2024. URL https://arxiv.org/abs/2410.10356

    2024 arXiv

  47. [55]

    Z. Ye, L. Chen, R. Lai, W. Lin, Y . Zhang, S. Wang, T. Chen, B. Kasikci, V . Grover, A. Krish- namurthy, et al. Flashinfer: Efficient and customizable attention engine for llm inference serving. arXiv preprint arXiv:2501.01005, 2025

    2025 arXiv

  48. [56]

    Z. Yuan, H. Zhang, P. Lu, X. Ning, L. Zhang, T. Zhao, S. Yan, G. Dai, and Y . Wang. Ditfastattn: Attention compression for diffusion transformer models,

  49. [57]

    Zaheer, G

    M. Zaheer, G. Guruganesh, A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed. Big bird: Transformers for longer sequences, 2021. URL https://arxiv.org/abs/2007.14062

    2021 arXiv

  50. [58]

    Zhang, B

    E. Zhang, B. Xiao, J. Tang, Q. Ma, C. Zou, X. Ning, X. Hu, and L. Zhang. Token pruning for caching better: 9 times acceleration on stable diffusion for free, 2024. URL https://arxiv.org/abs/2501.00375

    2024 arXiv

  51. [59]

    Zhang, J

    J. Zhang, J. wei, H. Huang, P. Zhang, J. Zhu, and J. Chen. Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration, 2025. URLhttps://arxiv.org/abs/2410.02367. 14

    2025

  52. [60]

    Zhang and K

    L. Zhang and K. Ma. Accelerating diffusion models with one-to-many knowledge distillation,

  53. [61]

    Zhang, Y

    P. Zhang, Y . Chen, R. Su, H. Ding, I. Stoica, Z. Liu, and H. Zhang. Fast video generation with sliding tile attention, 2025. URL https://arxiv.org/abs/2502.04507

    2025 arXiv

  54. [62]

    Zheng, X

    Z. Zheng, X. Peng, T. Yang, C. Shen, S. Li, H. Liu, Y . Zhou, T. Li, and Y . You. Open-sora: Democ- ratizing efficient video production for all, 2024. URLhttps://arxiv.org/abs/2412.20404

    2024 arXiv

  55. [63]

    C. Zou, E. Zhang, R. Guo, H. Xu, C. He, X. Hu, and L. Zhang. Accelerating diffusion transformers with dual feature caching, 2024. URLhttps://arxiv.org/abs/2412.18911

    2024

  56. [64]

    URL https://arxiv.org/abs/2410.04191

    arXiv

  57. [68]

    read" of the residual stream to compute scale and shift factors, (2) a distributive

    C. Zou, X. Liu, T. Liu, S. Huang, and L. Zhang. Accelerating diffusion transformers with token-wise feature caching, 2025. URLhttps://iclr.cc/virtual/2025/poster/27718. 15 A Extended Discussion of Latent Space Path Decompositions In this appendix, we formally demonstrate that ...

    2025

  58. [69]

    Attention and MLP layers output sums of scaled vectors, incrementally updating the residual stream

  59. [70]

    Modulations, scale and shift factors computed by layer normalization, and linear projections distribute over these vector additions

  60. [71]

    Non-distributive operations (softmax, GELU) are solely used to compute scalar coefficients of newly added vectors without directly modifying the residual stream. Thus, the latent space path of a DiT output at any inference stept, denoted zt, can be formally expressed as a sum ...

  61. [72]

    Compute sparse intermediate activations

  62. [73]

    Compute the difference against the cached activation

  63. [74]

    Multiply this sparse delta by the static value vectors (rows ofW2)

  64. [75]

    Directly accumulate this result into the cached output. This reduces computational overhead compared to the two-step subtraction-addition method required for dynamic attention vectors, but introduces additional challenges in kernel optimization. B.3.2 Persistent Grid and Warp-...

  65. [76]

    Persistent Grid Kernels:One threadblock is launched per GPU Streaming Multiprocessor (SM), allowing each threadblock to iterate over multiple work tiles. 19

  66. [77]

    This combination allows the overlap of the producer warpgroup’s memory loading prologue with the consumer warpgroups’ high latency epilogue operations

    Warp-Specialization: Within each threadblock, separate warp groups are assigned to compute/data loading operations, allowing better overlap between computation and memory operations. This combination allows the overlap of the producer warpgroup’s memory loading prologue with t...

  67. [78]

    With a torch-compiled bitpacking function, we reduce memory usage by 8x, while incurring negligible computational overhead

    Bitpacked Sparsity Masks: Standard boolean masks (torch.bool) consume one byte per entry. With a torch-compiled bitpacking function, we reduce memory usage by 8x, while incurring negligible computational overhead

  68. [79]

    anthropomorphic crow werecreature, photograph captured in a forest

    CPU Offloading with Double-Buffered Communication: We preallocate pinned (page-locked) CPU tensors and implement double-buffering on the GPU. This approach reduces GPU memory and communication overhead by overlapping GPU computations of the current layer with simultaneous tran...

  69. [2021]

    https://transformer-circuits.pub/2021/framework/index.html

    2021

  70. [2023]

    URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 33646ef0ed554145eab65f6250fab0c9-Paper-Conference.pdf

    2023

  71. [2024]

    URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/ 0267925e3c276e79189251585b4100bf-Abstract-Conference.html

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.