Pith. sign in

REVIEW 4 major objections 7 minor 36 cited by

Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity

T0 review · 4 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Video diffusion transformers can be accelerated more than two-fold without retraining by computing only the attention tokens that matter for each head.

desk verdict A credible training-free sparse-attention system for video DiTs with real wall-clock speedups; the main unresolved question is how far the 1% profiling proxy generalizes across prompts. read the letter →

arxiv 2502.01776 v2 pith:XKSBDS2Q submitted 2025-02-03 cs.CV cs.LG

Haocheng Xi , Shuo Yang , Yilong Zhao , Chenfeng Xu , Muyang Li , Xiuyu Li , Yujun Lin , Han Cai
show 6 more authors
Jintao Zhang Dacheng Li Jianfei Chen Ion Stoica Kurt Keutzer Song Han
This is my paper · ORCID
classification cs.CVcs.LG
keywords videodiffusiontransformerssparseattentionspatial-temporalsparsityheadclassificationtraining-freeaccelerationonlineprofilingtensorlayouttransformationinferencespeedup
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that three-dimensional full attention in video diffusion transformers is dominated by two structured sparse patterns: heads whose output is carried by tokens inside the same frame, and heads whose output is carried by tokens at the same spatial position in other frames. If that is true, then off-the-shelf video DiTs can be accelerated by over 2x with no retraining and with barely any change to the generated video, since only the important tokens need to be computed. The authors back this with an online profiler that samples 1% of query rows per head to decide which pattern to use, plus a layout transformation that turns the temporal pattern into a compact, tensor-core-friendly layout. The result they report is end-to-end speedups of 2.28x on CogVideoX-v1.5 and 2.33x on HunyuanVideo while holding PSNR near 30.

What carries the argument

The load-bearing machinery is the spatial/temporal head dichotomy together with the online profiling rule. Spatial Head and Temporal Head are the paper's names for two observed attention-map layouts: block-diagonal (same-frame tokens dominate) and slash-diagonal (same-position-across-frames tokens dominate). The profiler samples 1% of query rows per head, runs full, spatial, and temporal attention on those rows, and picks the sparse pattern with the lowest output MSE versus full attention; this converts the oracle pattern search into a ~3% overhead online decision. The layout transformation then transposes the activations from token-major to frame-major ordering, which makes the temporal head's noncontiguous access pattern contiguous and lets a block-sparse attention kernel reach near-theoretical speedup. Together these pieces carry the argument from observed sparsity to measured end-to-end speedup.

What would settle it

Run SVG on prompts with fast-moving objects or visible scene cuts, extract each head's profile-1% decision, and compare it to the oracle decision from full attention on the same step; if any head's choice flips or the generated PSNR falls below the reported ~29-30 on those prompts, the sampling proxy is falsified. A simpler quantitative check is to measure the agreement rate between profile-1% and profile-100% classifications across heads and denoising steps and identify heads where the MSE ranking reverses.

Watch

Extended reading notes

Core claim

SVG claims that every attention head in a video diffusion transformer is, at each denoising step, effectively either a Spatial Head (only tokens within the same frame contribute meaningfully, giving a block-diagonal attention map) or a Temporal Head (only tokens at the same spatial location across frames contribute, giving a slash-diagonal map). The method classifies each head cheaply: it computes full, spatial-masked, and temporal-masked attention on a random 1% of query rows, and selects the sparse pattern whose output has lower mean squared error against full attention. It then executes only that pattern, reordering the temporal head's strided token access into a contiguous frame-major layout so GPU tensor cores can use it. On this basis the paper reports preserving generation quality (PSNR above 29) while cutting attention FLOPS to roughly 30%, yielding end-to-end speedups of 2.28x, 2.33x, and 1.51x on CogVideoX-v1.5, HunyuanVideo, and Wan 2.1, with an additional ~1.3x from FP8 quantization on HunyuanVideo.

Load-bearing premise

The claim rests on 1% of randomly sampled query rows per head being enough to reveal whether that head is spatial or temporal; if for some head, step, or prompt those rows do not show the head's true dominant pattern, the chosen sparse attention will deviate from full attention and reported quality will drop.

Editorial extensions

If this is right

  • The reported 2.28x on CogVideoX-v1.5 and 2.33x on HunyuanVideo mean a minutes-long video generation can be cut roughly in half with no retraining and no visible quality loss; with FP8 the HunyuanVideo run drops from 2253s to 968s.
  • Because the choice of spatial versus temporal pattern is per-head and per-step, the method tracks the dynamic changes in attention during denoising instead of relying on a fixed schedule.
  • The method composes with orthogonal acceleration techniques: the paper shows FP8 attention adds another ~1.3x throughput at a 0.1 PSNR cost on HunyuanVideo.
  • The same pipeline on Wan 2.1 reaches 1.51x speedup, indicating the spatial/temporal dichotomy is not specific to one model family.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 1%-sample profiling rule is a bet that a tiny query subset reveals the head's global attention structure; a direct test would be to run SVG on prompts with fast motion or scene cuts, where the same-position temporal correspondence breaks down, and to make the sampling ratio motion-aware.
  • The spatial/temporal head splitting may transfer to other spatiotemporal transformers, such as video-language encoders, where the same three-dimensional attention layout appears and can be profiled in the same way.
  • The paper fixes the sparsity ratios cs and ct manually; a natural extension is to tie them to per-scene content such as motion magnitude or resolution, or to cache head types across nearby denoising steps to reduce the 3% profiling overhead further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes Sparse VideoGen (SVG), a training-free framework for accelerating video diffusion transformers by exploiting sparsity in 3D full attention. It classifies attention heads into spatial and temporal types via an online profiling strategy that samples 1% of query rows, uses a hardware-efficient layout transformation to make temporal-head sparsity tensor-core friendly, and adds customized kernels and optional FP8 quantization. On CogVideoX-v1.5-I2V, CogVideoX-v1.5-T2V, and HunyuanVideo-T2V, the authors report end-to-end speedups of up to 2.28x and 2.33x (with FP8) while maintaining PSNR values around 28-30, outperforming baselines such as DiTFastAttn, MInference, and PAB. The code is open-sourced.

Significance. If the reported results hold, SVG is a practical, training-free acceleration method for video DiTs, with the key novelty being the online per-head classification into spatial and temporal patterns and the layout transformation that turns a hardware-hostile strided pattern into a contiguous block-sparse one. The paper's strengths include end-to-end wall-clock measurements on H100 across multiple models, a clear ablation of the layout transformation (Figure 8) showing that the system achieves near-theoretical speedups, and open-sourced code that supports reproducibility. The main uncertainty is whether the 1% profiling assumption is robust across diverse prompts and motions, which the current aggregate metrics do not yet establish.

major comments (4)
  1. [Section 4.1, Algorithm 1, Table 3] The central quality-preservation claim rests on the assumption that per-head masks selected from 1% of randomly sampled query rows match the oracle (100%) selection for every head, denoising step, and prompt. Table 3 provides only aggregate PSNR/SSIM/LPIPS on a random subset of VBench for CogVideoX-I2V, which does not rule out misclassification of rare-but-important tokens (e.g., fast-moving objects or rare interactions) that are unlikely to appear in the 1% sample. Since there is no fallback to dense attention when the sampled MSE is ambiguous, and errors can accumulate over the roughly 50 denoising steps, the claim that SVG 'preserves generation quality' is not fully established. Please add per-head/step mask-agreement rates against the 100%-profiling oracle, or stratify the evaluation by motion/prompt difficulty, or report worst-case (e.g., 5th percentile) PSNR across the benchmark.
  2. [Section 5.2, Tables 1, 3, 4] The quality metrics that support the 'preserving generation quality' claim are reported as point estimates without error bars, confidence intervals, or significance tests. For example, Table 3 shows a difference of 0.206 PSNR between the 1% and 100% profiling ratios; without the number of prompts, random seed, or run-to-run variance, it is unclear whether this difference is meaningful or within noise. Please report the number of prompts, the random seed, and error bars (or per-prompt distributions) for the core quality metrics.
  3. [Section 3.1, Section 5.1, Algorithm 1] The temporal-head pattern is described inconsistently with the configuration parameters. Section 3.1 states that a temporal head attends to tokens at the same spatial position across frames (a slash-wise pattern with stride L), which would involve about N tokens per query (e.g., 11 for CogVideoX), but Section 5.1 sets ct = 1224 for CogVideoX and ct = 1200 for HunyuanVideo, orders of magnitude larger than N. The exact definitions of gen_spatial_mask() and gen_temporal_mask() in Algorithm 1 are never given, and this ambiguity propagates to the theoretical analysis in Section 3.3. Please provide the precise mask construction and reconcile the narrative with the parameter choices.
  4. [Section 3.3, Table 4, Figure 8] The paper uses 'sparsity' to mean the fraction of full attention computation retained (i.e., density), which is the opposite of the standard usage in the sparse-attention literature, where sparsity denotes the fraction of tokens skipped. For example, Section 3.3 says 'the sparsity can easily achieve 30%' while the configuration computes about 30% of the full-attention FLOPs (so 70% is skipped). This inverted terminology makes Table 4 and Figure 8 difficult to interpret (e.g., LPIPS appears to improve as 'sparsity' increases). Please rename the quantity to 'density' or 'computation ratio' throughout and update the text, table, and figure axes accordingly.
minor comments (7)
  1. [Abstract / Section 5] The abstract reports a 1.51x speedup on Wan 2.1, but no quantitative Wan 2.1 results appear in the experiments section; the appendix only shows visual comparisons. Please add the quantitative benchmark or remove the claim from the abstract.
  2. [Section 3.2] The title 'Sparse attention achieves lossless accuracy' overstates the evidence: the reported PSNR values (28-30) and the approximation in Table 3 are not strictly lossless. Suggest rewording to 'near-lossless' or 'high-fidelity'.
  3. [Algorithm 1] The comment 't: Sampled Token Number. E.g., 32' is inconsistent with the 1% profiling ratio used in the experiments (which would be roughly 448 tokens for CogVideoX). Please clarify how t is set in practice.
  4. [Section 4.1] The statement that profiling adds only 3% runtime overhead is not backed by a table or figure; please provide the measurement or point to an appendix entry.
  5. [Figure 8] The speedup annotations in Figure 8 overlap and are difficult to read; please reposition them for clarity.
  6. [Section 5.1] The phrase 'We skip the first 25% denoising steps for all baselines' is ambiguous about whether the dense baseline is also affected; please clarify that this applies to the accelerated baselines only.
  7. [Section 3.1 and Appendix B] There are minor typos: 'instinct sparsity' in the Section 3.1 heading should be 'intrinsic sparsity', and 'Comparion' in the Appendix B captions should be 'Comparison'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: speedup is measured wall-clock time and quality is measured against the dense model's own outputs, with the 1% profiling proxy directly validated against the 100% oracle in Table 3.

full rationale

The paper's central claims are end-to-end latency speedups and generation-quality preservation measured by PSNR, SSIM, LPIPS, and VBench metrics computed against the dense full-attention model's outputs, not against any quantity that the method fits. The online profiling strategy (Section 4.1, Algorithm 1) samples 1% of query rows to choose between spatial and temporal masks, but Table 3 explicitly compares the resulting generation quality at profiling ratios 0.1%, 1%, 5%, and 100%, showing that the 1% proxy is empirically validated against the oracle rather than assumed equivalent by construction. The sparsity hyperparameters cs, ct, and the profiling ratio are hand-tuned, but they do not define the success metric, so tuning them is not circular. The spatial-head/temporal-head taxonomy is introduced as an empirical observation from attention-map inspection (Figure 3), not as a theorem derived from the method's own outputs, and no uniqueness result is imported from self-citations. Citations to prior work with overlapping authorship, such as DuoAttention, are used only to support the general phenomenon of head-diverse sparsity and are not load-bearing for the claimed speedup or quality numbers. No equation or fitted parameter is renamed as a prediction, and no claim reduces to its inputs by definition.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

Central claim rests on two empirical observations (head patterns are spatial/temporal; 1% sampling classifies correctly) and on hand-chosen sparsity hyperparameters (cs, ct) and profiling ratio. No new physical entities are introduced.

free parameters (4)
  • cs (spatial window frames) = 4 (CogVideoX), 10 (HunyuanVideo)
    Window of frames each spatial head attends to; chosen to reach about 30% sparsity while keeping PSNR high (Sec 5.1).
  • ct (temporal tokens) = 1224 (CogVideoX), 1200 (HunyuanVideo)
    Number of temporally matched tokens each temporal head attends to; hand-selected for quality/efficiency tradeoff (Sec 5.1).
  • profiling_ratio = 1%
    Fraction of query rows sampled for online head classification; chosen from sensitivity test (Table 3) as the smallest ratio with quality near the oracle.
  • denoising_step_skip = 25%
    First 25% of denoising steps run with full attention for all methods; protocol adopted from prior work (Sec 5.1).
assumptions (4)
  • standard math FlashAttention associativity justifies layout transformation.
    Sec 4.2: 'attention computation is associative (Dao et al., 2022; 2019)'.
  • domain assumption Video training data is mostly slow motion, so temporal heads only need same-position tokens across frames.
    Footnote 1 in Sec 3.1 states this as a hypothesis for the temporal head pattern.
  • domain assumption Attention outputs are dominated by either spatially local or temporally aligned tokens; text prompt and first frame tokens are important for both.
    Sec 3.1 and Figure 3 present this as an empirical observation across models.
  • domain assumption 1% sampled queries are representative for head classification.
    Sec 4.1 and Table 3 justify this empirically, but it is not proven for all heads, steps, and prompts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity." pith.science (2026). https://pith.science/paper/XKSBDS2Q

@misc{pith2026250201776,
  author       = {Pith},
  title        = {Pith review of: Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XKSBDS2Q}},
  note         = {Machine review of arXiv:2502.01776}
}
read the original abstract

Diffusion Transformers (DiTs) dominate video generation but their high computational cost severely limits real-world applicability, usually requiring tens of minutes to generate a few seconds of video even on high-performance GPUs. This inefficiency primarily arises from the quadratic computational complexity of 3D Full Attention with respect to the context length. In this paper, we propose a training-free framework termed Sparse VideoGen (SVG) that leverages the inherent sparsity in 3D Full Attention to boost inference efficiency. We reveal that the attention heads can be dynamically classified into two groups depending on distinct sparse patterns: (1) Spatial Head, where only spatially-related tokens within each frame dominate the attention output, and (2) Temporal Head, where only temporally-related tokens across different frames dominate. Based on this insight, SVG proposes an online profiling strategy to capture the dynamic sparse patterns and predicts the type of attention head. Combined with a novel hardware-efficient tensor layout transformation and customized kernel implementations, SVG achieves up to 2.28x and 2.33x end-to-end speedup on CogVideoX-v1.5 and HunyuanVideo, respectively, while preserving generation quality. Our code is open-sourced and is available at https://github.com/svg-project/Sparse-VideoGen

Figures

Figures reproduced from arXiv: 2502.01776 by the authors.

Figure 1
Figure 1. SVG accelerates video generation while maintaining high quality. On CogVideoX-v1.5-I2V and Hunyuan-T2V, our method achieves a 2.28× and 2.33× speedup with high PSNR. In contrast, MInference (Jiang et al., 2024) fails to maintain pixel fidelity (significant blurring in the first example) and temporal coherence (inconsistencies in the tree trunk in the second example). tion. This success has been naturally carried ove… view at source ↗
Figure 2
Figure 2. Attention dominates the computation in video diffu￾sion models. For CogVideoX-v1 and -v1.5 with 17k and 45k context length, attention takes 51% and 73% of the latency, respec￾tively. For HunyuanVideo with 120k context length, attention can take over 80% amount of the runtime latency. from animation (Guo et al., 2024; Feng et al., 2024) to physical world simulation (Liu et al., 2024b). Despite significant advances in… view at source ↗
Figure 3
Figure 3. We observe two types of attention maps with distinct sparse patterns: spatial map (b) and temporal map (d). Based on the attention map, we classify all attention heads into Spatial Head (a) and Temporal Head (c), which contribute to the spatial and temporal consistency of generated videos respectively. As visualized in (e), spatial head primarily focuses on all tokens within the same frame (painted as red). In contr… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Overview of SVG framework. (a) During generation, SVG adaptively classifies each attention head as either a spatial head or a temporal head and applies a dedicated sparse attention computation accordingly. (b) This adaptive classification is driven by online profiling …
Figure 5
Figure 5. Figure 5: Visualization of hardware-efficient layout transforma￾tion. (a) Non-contiguous sparsity layout of temporal head, which is hardware inefficient due to the contiguous layout required by hardware accelerators. (b) Contiguous layout generated by trans￾posing the token-majo…
Figure 6
Figure 6. Figure 6 [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: The breakdown of end-to-end runtime of HunyuanVideo when generating a 5.3s, 720p video. SVG effectively reduces the end-to-end inference time from 2253 seconds to 968 seconds through system-algorithm co-design. Each design point contributes to a considerable improvemen…
Figure 8
Figure 8. Figure 8: Latency comparison of different implementations of sparse attention. Our hardware-efficient layout transformation optimizes the sparsity pattern of temporal head for better con￾tiguity, which is 1.7× faster than naive sparse attention (named original), approaching the …
Figure 9
Figure 9. Figure 9: Comparion of Dense Attention and Sparse VideoGen on HunyuanVideo Text-to-Video generation. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Comparion of Dense Attention and Sparse VideoGen on Wan 2.1 Text-to-Video generation. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Comparison of Dense Attention and Sparse VideoGen on Wan 2.1 Image-to-Video generation. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 36 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation

    cs.CV 2026-02 conditional novelty 7.0 of 10

    Causal Forcing uses an autoregressive teacher for ODE initialization in diffusion distillation to close the causal attention gap and deliver better real-time video generation than Self Forcing.

  2. FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Models

    cs.CV 2025-09 conditional novelty 7.0 of 10

    FG-Attn accelerates video DiT inference by computing attention over only the relevant Mx1 key slices per query group, using an asynchronous gather-load kernel, and reports up to 1.65x end-to-end speedup with negligibl...

  3. Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers

    cs.CV 2026-07 conditional novelty 6.5 of 10

    A hybrid NoPE visual diffusion backbone with HeteroP scaling yields ~7× pretraining compute efficiency versus matched full attention and near-stable zero-shot 5s→30s video extrapolation.

  4. ContextMaster: Interactive Multi-Shot Video Creation via Fixed-Budget Sparse Context Routing

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A unified video model with role-aware context and fixed-budget sparse attention enables interactive multi-shot generation, reference conditioning, and editing at 16 FPS on a single GPU.

  5. Token Radius Attention for Efficient Video Generation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Video diffusion transformers can run ~1.5-2x faster with competitive quality by converting each query's attention entropy into a spatially decayed retention radius instead of dense attention.

  6. EchoCache: Energy-Guided Cross-Modal Caching for Efficient Audio-Driven Video Generation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Audio time-frequency energy guides which video latents get recomputed during diffusion denoising, yielding up to 2.46x faster audio-driven video generation with competitive quality.

  7. SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A 5B/14B video diffusion transformer using 75% linear + 25% softmax attention with block attention residuals generates 480p/720p video on one GPU faster than full-softmax models at comparable VBench quality.

  8. Surprise Forcing: What to Remember, When to Skip in Long Video Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A training-free 'surprise' controller decides which old frames to keep in memory and which chunks need fewer denoising steps, improving long-video consistency at real-time speed.

  9. DiTango: Cost-Effective Parallel Diffusion Generation with Selective Attention State Reuse

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Selective reuse of composed attention states across denoising steps lets DiTango skip both remote KV communication and attention compute for low-contribution sequence partitions, cutting multi-GPU diffusion latency by...

  10. Kaleido: Algorithm-Hardware Co-Design for Video Diffusion Transformers by Exploiting Latent Space Correlations

    cs.AR 2026-07 conditional novelty 6.0 of 10

    A channel-wise reuse algorithm plus a reconfigurable systolic accelerator skips redundant vDiT attention and MLP computation, achieving up to 5.9x speedup and 16x energy savings.

  11. Controlling Motion Transfer in Diffusion Transformers via Attention Heads

    cs.CV 2026-07 accept novelty 6.0 of 10

    Video DiTs encode motion and structure in separate attention-head subsets; selecting and guiding those heads yields training-free motion transfer with higher fidelity and structural alignment than existing methods.

  12. SAF3R: Dynamic Sparse Attention for Feed-Forward 3D Reconstruction Transformers

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Training-free head-wise dynamic sparse attention preserves F3R pose and reconstruction quality while delivering multi-fold end-to-end speedups on long image sequences.

  13. Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attention

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Sparse attention with chunk-aware sparsity growth and hierarchical frame/block selection accelerates autoregressive video diffusion at ~1.3x with VBench quality on par with dense attention.

  14. Transition Matching Distillation for Fast Video Generation

    cs.CV 2026-01 conditional novelty 6.0 of 10

    Splitting a video diffusion model into a fixed feature extractor and a small recurrent flow head lets TMD generate videos in one to two effective steps with better VBench scores than prior distilled models.

  15. Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformers

    cs.CV 2026-01 conditional novelty 6.0 of 10

    MOD-DiT accelerates video DiTs by extrapolating fitted attention-pattern intensities across denoising steps to build dynamic block masks, achieving 1.8–2.3× speedups on CogVideoX, HunyuanVideo, and Wan2.1.

  16. SnapGen++: Unleashing Diffusion Transformers for Efficient High-Fidelity Image Generation on Edge Devices

    cs.CV 2026-01 conditional novelty 6.0 of 10

    A compact elastic diffusion transformer with adaptive sparse attention and knowledge-guided distribution-matching distillation achieves 4-step 1K image generation on a phone in roughly 1.8 seconds.

  17. TinyHistory: Lightweight Video History Embeddings via Two-Stage Context Learning

    cs.CV 2025-12 conditional novelty 6.0 of 10

    TinyHistory compresses long video history into a ~5k-token context via a two-stage learning scheme, achieving consistency on par with heavier baselines at lower memory cost.

  18. Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers

    cs.CV 2025-12 conditional novelty 6.0 of 10

    LLSA uses hierarchical top-K selection plus coarse-token enrichment to make diffusion-transformer attention O(N log N), giving about 6x faster training at 256x256 pixel resolution with FID comparable to full attention.

  19. End-to-End Training for Autoregressive Video Diffusion via Self-Resampling

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Resampling Forcing trains autoregressive video diffusion models on self-resampled degraded histories with a causal mask, achieving stable long-horizon generation without a teacher or discriminator.

  20. Timeripple: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Space

    cs.AR 2025-11 conditional novelty 6.0 of 10

    Timeripple cuts vDiT self-attention compute by up to 85% by reusing partial attention scores of spatially and temporally correlated tokens across channels, with VBench quality essentially unchanged.

  21. OmniCache: A Trajectory-Oriented Global Perspective on Training-Free Cache Reuse for Diffusion Transformer Models

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    A training-free cache-reuse scheme that spreads computation across the full diffusion trajectory and subtracts estimated noise, accelerating DiT sampling with claimed competitive quality.

  22. Waver: Wave Your Way to Lifelike Video Generation

    cs.CV 2025-08 conditional novelty 6.0 of 10

    Waver unifies text-to-video, image-to-video, and text-to-image generation in a single 12B-parameter DiT with a hybrid dual/single-stream architecture and a cascade refiner, claiming top-three public leaderboard performance.

  23. FreeLong++: Training-Free Long Video Generation via Multi-band SpectralFusion

    cs.CV 2025-06 conditional novelty 6.0 of 10

    FreeLong++ extends short-video diffusion models to 4x to 8x longer clips, without retraining, by fusing multiple windowed attention branches through frequency-domain filters and a spectral noise initialization.

  24. VMoBA: Mixture-of-Block Attention for Video Diffusion Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    VMoBA is a sparse attention mechanism for video diffusion models that combines cyclic 1D-2D-3D block partitioning with global and threshold-based block selection to reduce training FLOPs while keeping generation quality.

  25. TurboVSR: Fantastic Video Upscalers and Where to Find Them

    cs.CV 2025-06 conditional novelty 6.0 of 10

    TurboVSR uses a high-compression video autoencoder with factorized conditioning and non-uniform shortcut sampling to achieve near-state-of-the-art perceptual video super-resolution at roughly 100x lower compute cost.

  26. Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Chipmunk speeds up diffusion transformer inference by recomputing, at each step, only the top percent of attention and MLP activation columns that change most between steps, caching the rest in column-sparse GPU kernels.

  27. Dual-Expert Consistency Model for Efficient and High-Quality Video Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    By training a semantic expert and a LoRA-based detail expert, DCM reaches nearly teacher-level VBench scores with 4-step video sampling on HunyuanVideo and CogVideoX.

  28. SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference

    cs.CV 2026-08 conditional novelty 5.0 of 10

    SPADE combines static, semi-static, and dynamic block-sparse attention with a cheap SICS-based blocking heuristic to speed up video diffusion inference by up to 1.80x end-to-end.

  29. ACID: Adaptive Caching for vIDeo generation

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Adaptive threshold switching on the drift signal's rate of change expands the quality-vs-speed Pareto frontier of TeaCache, EasyCache, and DiCache across HunyuanVideo, Wan 2.1, and CogVideoX.

  30. MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models

    cs.AI 2026-06 conditional novelty 5.0 of 10

    By sampling variance-inflated query vectors during prefilling, MM-ShiftKV selects prompt KV caches that better match decoding-time attention and outperforms prior prefill-only KV compression on multimodal benchmarks a...

  31. UltraImageGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attention

    cs.CV 2025-10 conditional novelty 5.0 of 10

    A pretrained FLUX diffusion model is adapted with local-window attention plus low-resolution global guidance, allowing 4K text-to-image generation from 1K-only training data at about 2x lower cost.

  32. NABLA: Neighborhood Adaptive Block-Level Attention

    cs.CV 2025-07 conditional novelty 5.0 of 10

    NABLA builds content-aware block-sparse attention masks by pooling queries and keys and thresholding their softmax CDF, giving near-baseline video quality at up to 2.7x inference speedup and 1.46x pretraining speedup.

  33. Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Sparse-vDiT replaces dense attention with fixed per-head sparse patterns chosen offline, achieving 1.58-1.85x end-to-end speedups on CogVideoX1.5, HunyuanVideo, and Wan2.1 with minimal quality loss.

  34. RainFusion: Adaptive Video Generation Acceleration via Multi-Dimensional Visual Redundancy

    cs.CV 2025-05 conditional novelty 5.0 of 10

    Training-free sparse attention that classifies each head as spatial, temporal, or textural and applies a matched mask or token reduction, giving about 1.9x attention speedup with small VBench losses.

  35. PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models

    cs.CV 2025-06 conditional novelty 4.0 of 10

    PAROAttention permutes tokens along frame, height, and width axes to make visual attention block-wise, enabling sparse and INT8/INT4 quantized attention with near-baseline generation quality.

  36. SageAttention2++: A More Efficient Implementation of SageAttention2

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Using the FP16-accumulator FP8 matmul instruction with narrowed quantization ranges for P and V yields up to 3.9x speedup over FlashAttention2 with negligible accuracy loss.

Reference graph

Works this paper leans on

71 extracted references · 25 canonical work pages · cited by 36 Pith papers

  1. [1]

    Vivit: A video vision transformer

    Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lu c i \'c , M., and Schmid, C. Vivit: A video vision transformer. In ICCV, 2021

    work page 2021

  2. [2]

    Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction

    Cai, H., Li, J., Hu, M., Gan, C., and Han, S. Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 17302--17313, 2023

    work page 2023

  3. [3]

    Condition-aware neural network for controlled image generation

    Cai, H., Li, M., Zhang, Q., Liu, M.-Y., and Han, S. Condition-aware neural network for controlled image generation. In CVPR, 2024

    work page 2024

  4. [4]

    Deep compression autoencoder for efficient high-resolution diffusion models

    Chen, J., Cai, H., Chen, J., Xie, E., Yang, S., Tang, H., Li, M., Lu, Y., and Han, S. Deep compression autoencoder for efficient high-resolution diffusion models. arXiv preprint arXiv:2410.10733, 2024 a

    arXiv 2024

  5. [5]

    Pixart-sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation

    Chen, J., Ge, C., Xie, E., Wu, Y., Yao, L., Ren, X., Wang, Z., Luo, P., Lu, H., and Li, Z. Pixart-sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision, pp.\ 74--91. Springer, 2025

    work page 2025

  6. [6]

    delta -dit: A training-free acceleration method tailored for diffusion transformers

    Chen, P., Shen, M., Ye, P., Cao, J., Tu, C., Bouganis, C.-S., Zhao, Y., and Chen, T. delta -dit: A training-free acceleration method tailored for diffusion transformers. arXiv preprint arXiv:2406.01125, 2024 b

    arXiv 2024

  7. [7]

    Rethinking attention with performers

    Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020

    arXiv 2009

  8. [8]

    Learning fast algorithms for linear transforms using butterfly factorizations

    Dao, T., Gu, A., Eichhorn, M., Rudra, A., and R \'e , C. Learning fast algorithms for linear transforms using butterfly factorizations. In International conference on machine learning, pp.\ 1517--1527. PMLR, 2019

    work page 2019

Show all 71 references
  1. [9]

    Y., Ermon, S., Rudra, A., and Ré, C

    Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. URL https://arxiv.org/abs/2205.14135

    2022 arXiv

  2. [10]

    J., and Zhang, X

    Feng, H., Ding, Z., Xia, Z., Niklaus, S., Abrevaya, V., Black, M. J., and Zhang, X. Explorative inbetweening of time and space. In European Conference on Computer Vision, pp.\ 378--395. Springer, 2024

    2024

  3. [11]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Guo, Y., Yang, C., Rao, A., Liang, Z., Wang, Y., Qiao, Y., Agrawala, M., Lin, D., and Dai, B. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In ICLR, 2024

    2024

  4. [12]

    Lm-infinite: Simple on-the-fly length generalization for large language models

    Han, C., Wang, Q., Xiong, W., Chen, Y., Ji, H., and Wang, S. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137, 2023

    2023 arXiv

  5. [13]

    Denoising diffusion probabilistic models

    Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. NeurIPS, 2020

    2020

  6. [14]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Hong, W., Ding, M., Zheng, W., Liu, X., and Tang, J. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. In ICLR, 2023

    2023

  7. [15]

    and Ziou, D

    Horé, A. and Ziou, D. Image quality metrics: Psnr vs. ssim. In 2010 20th International Conference on Pattern Recognition, pp.\ 2366--2369, 2010. doi:10.1109/ICPR.2010.579

    2010 doi

  8. [16]

    Vbench: Comprehensive benchmark suite for video generative models, 2023

    Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., Wang, Y., Chen, X., Wang, L., Lin, D., Qiao, Y., and Liu, Z. Vbench: Comprehensive benchmark suite for video generative models, 2023. URL https://arxiv.org/abs/2311.17982

    2023 arXiv

  9. [17]

    H., Li, D., Lin, C.-Y., et al

    Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. arXiv preprint arXiv:2407.02490, 2024

    2024 arXiv

  10. [18]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp.\ 5156--5165. PMLR, 2020

    2020

  11. [19]

    Streamdiffusion: A pipeline-level solution for real-time interactive generation

    Kodaira, A., Xu, C., Hazama, T., Yoshimoto, T., Ohno, K., Mitsuhori, S., Sugano, S., Cho, H., Liu, Z., and Keutzer, K. Streamdiffusion: A pipeline-level solution for real-time interactive generation. arXiv preprint arXiv:2312.12491, 2023

    2023 arXiv

  12. [20]

    Hunyuanvideo: A systematic framework for large video generative models

    Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Wu, B., Zhang, J., et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  13. [21]

    Distrifusion: Distributed parallel inference for high-resolution diffusion models

    Li, M., Cai, T., Cao, J., Zhang, Q., Cai, H., Bai, J., Jia, Y., Liu, M.-Y., Li, K., and Han, S. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In CVPR, 2024

    2024

  14. [22]

    Svdquant: Absorbing outliers by low-rank components for 4-bit diffusion models

    Li*, M., Lin*, Y., Zhang*, Z., Cai, T., Li, X., Guo, J., Xie, E., Meng, C., Zhu, J.-Y., and Han, S. Svdquant: Absorbing outliers by low-rank components for 4-bit diffusion models. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  15. [23]

    Q-diffusion: Quantizing diffusion models

    Li, X., Liu, Y., Lian, L., Yang, H., Dong, Z., Kang, D., Zhang, S., and Keutzer, K. Q-diffusion: Quantizing diffusion models. In ICCV, 2023

    2023

  16. [24]

    H., Li, D., Gao, J., Yang, Y., et al

    Li, Y., Jiang, H., Zhang, C., Wu, Q., Luo, X., Ahn, S., Abdi, A. H., Li, D., Gao, J., Yang, Y., et al. Mminference: Accelerating pre-filling for long-context vlms via modality-aware permutation sparse attention. arXiv preprint arXiv:2504.16083, 2025

    2025 arXiv

  17. [25]

    Looking backward: Streaming video-to-video translation with feature banks

    Liang, F., Kodaira, A., Xu, C., Tomizuka, M., Keutzer, K., and Marculescu, D. Looking backward: Streaming video-to-video translation with feature banks. arXiv preprint arXiv:2405.15757, 2024

    2024 arXiv

  18. [26]

    Timestep embedding tells: It's time to cache for video diffusion model

    Liu, F., Zhang, S., Wang, X., Wei, Y., Qiu, H., Zhao, Y., Zhang, Y., Ye, Q., and Wan, F. Timestep embedding tells: It's time to cache for video diffusion model. arXiv preprint arXiv:2411.19108, 2024 a

    2024 arXiv

  19. [27]

    World model on million-length video and language with ringattention

    Liu, H., Yan, W., Zaharia, M., and Abbeel, P. World model on million-length video and language with ringattention. arXiv preprint, 2024 b

    2024

  20. [28]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022

    2022 arXiv

  21. [29]

    Instaflow: One step is enough for high-quality diffusion-based text-to-image generation

    Liu, X., Zhang, X., Ma, J., Peng, J., and Liu, Q. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. In International Conference on Learning Representations, 2024 c

    2024

  22. [30]

    Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time

    Liu, Z., Desai, A., Liao, F., Wang, W., Xie, V., Xu, Z., Kyrillidis, A., and Shrivastava, A. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems, 36, 2024 d

    2024

  23. [31]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps

    Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In NeurIPS, 2022 a

    2022

  24. [32]

    Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models

    Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022 b

    2022 arXiv

  25. [33]

    Latent consistency models: Synthesizing high-resolution images with few-step inference

    Luo, S., Tan, Y., Huang, L., Li, J., and Zhao, H. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv: 2310.04378, 2023

    2023 arXiv

  26. [34]

    Lv, Z., Si, C., Song, J., Yang, Z., Qiao, Y., Liu, Z., and Wong, K.-Y. K. Fastercache: Training-free video diffusion model acceleration with high quality. 2024

    2024

  27. [35]

    Deepcache: Accelerating diffusion models for free

    Ma, X., Fang, G., and Wang, X. Deepcache: Accelerating diffusion models for free. In CVPR, 2024

    2024

  28. [36]

    SDE dit: Guided image synthesis and editing with stochastic differential equations

    Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. SDE dit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022

    2022

  29. [37]

    and Xie, S

    Peebles, W. and Xie, S. Scalable diffusion models with transformers. In ICCV, 2023

    2023

  30. [38]

    J., Soran, B., Krishnamoorthi, R., Elhoseiny, M., and Chandra, V

    Shen, X., Xiong, Y., Zhao, C., Wu, L., Chen, J., Zhu, C., Liu, Z., Xiao, F., Varadarajan, B., Bordes, F., Liu, Z., Xu, H., Kim, H. J., Soran, B., Krishnamoorthi, R., Elhoseiny, M., and Chandra, V. Longvu: Spatiotemporal adaptive compression for long video-language understandin...

    2024 arXiv

  31. [39]

    Denoising diffusion implicit models

    Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In ICLR, 2020

    2020

  32. [40]

    and Ermon, S

    Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019

    2019

  33. [41]

    Consistency models

    Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consistency models. In ICML, 2023

    2023

  34. [42]

    Akvq-vl: Attention-aware kv cache adaptive 2-bit quantization for vision-language models, 2025

    Su, Z., Shen, W., Li, L., Chen, Z., Wei, H., Yu, H., and Yuan, K. Akvq-vl: Attention-aware kv cache adaptive 2-bit quantization for vision-language models, 2025. URL https://arxiv.org/abs/2501.15021

    2025 arXiv

  35. [43]

    Quest: Query-aware sparsity for efficient long-context llm inference, 2024

    Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikci, B., and Han, S. Quest: Query-aware sparsity for efficient long-context llm inference, 2024. URL https://arxiv.org/abs/2406.10774

    2024 arXiv

  36. [44]

    Tillet, P., Kung, H.-T., and Cox, D. D. Triton: an intermediate language and compiler for tiled neural network computations. Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, 2019. URL https://api.semanticscholar.org/Corpu...

    2019

  37. [45]

    Wan: Open and advanced large-scale video generative models

    Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.-W., Chen, D., Yu, F., Zhao, H., Yang, J., Zeng, J., et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025

    2025 arXiv

  38. [46]

    Z., Khabsa, M., Fang, H., and Ma, H

    Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020

    2006 arXiv

  39. [48]

    Duoattention: Efficient long-context llm inference with retrieval and streaming heads

    Xiao, G., Tang, J., Zuo, J., Guo, J., Yang, S., Tang, H., Fu, Y., and Han, S. Duoattention: Efficient long-context llm inference with retrieval and streaming heads. arXiv preprint arXiv:2410.10819, 2024 a

    2024 arXiv

  40. [49]

    Efficient streaming language models with attention sinks, 2024 b

    Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks, 2024 b . URL https://arxiv.org/abs/2309.17453

    2024 arXiv

  41. [50]

    Sana: Efficient high-resolution image synthesis with linear diffusion transformers

    Xie, E., Chen, J., Chen, J., Cai, H., Tang, H., Lin, Y., Zhang, Z., Li, M., Zhu, L., Lu, Y., et al. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. arXiv preprint arXiv:2410.10629, 2024

    2024 arXiv

  42. [51]

    Xattention: Block sparse attention with antidiagonal scoring

    Xu, R., Xiao, G., Huang, H., Guo, J., and Han, S. Xattention: Block sparse attention with antidiagonal scoring. arXiv preprint arXiv:2503.16428, 2025

    2025 arXiv

  43. [52]

    Tidaldecode: Fast and accurate llm decoding with position persistent sparse attention

    Yang, L., Zhang, Z., Chen, Z., Li, Z., and Jia, Z. Tidaldecode: Fast and accurate llm decoding with position persistent sparse attention. arXiv preprint arXiv:2410.05076, 2024 a

    2024 arXiv

  44. [53]

    E., Stoica, I., and Zheng, L

    Yang, S., Sheng, Y., Gonzalez, J. E., Stoica, I., and Zheng, L. Post-training sparse attention with double sparsity, 2024 b . URL https://arxiv.org/abs/2408.07092

    2024 arXiv

  45. [54]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng, G., et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024 c

    2024 arXiv

  46. [55]

    Sparsetir: Composable abstractions for sparse compilation in deep learning, 2023

    Ye, Z., Lai, R., Shao, J., Chen, T., and Ceze, L. Sparsetir: Composable abstractions for sparse compilation in deep learning, 2023. URL https://arxiv.org/abs/2207.04606

    2023 arXiv

  47. [56]

    Flashinfer: Efficient and customizable attention engine for llm inference serving, 2025

    Ye, Z., Chen, L., Lai, R., Lin, W., Zhang, Y., Wang, S., Chen, T., Kasikci, B., Grover, V., Krishnamurthy, A., and Ceze, L. Flashinfer: Efficient and customizable attention engine for llm inference serving, 2025. URL https://arxiv.org/abs/2501.01005

    2025 arXiv

  48. [57]

    Yin, T., Gharbi, M., Park, T., Zhang, R., Shechtman, E., Durand, F., and Freeman, W. T. Improved distribution matching distillation for fast image synthesis. arXiv preprint arXiv:2405.14867, 2024 a

    2024 arXiv

  49. [58]

    T., and Park, T

    Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., and Park, T. One-step diffusion with distribution matching distillation. In CVPR, 2024 b

    2024

  50. [59]

    Metaformer is actually what you need for vision

    Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., and Yan, S. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10819--10829, 2022

    2022

  51. [60]

    Ditfastattn: Attention compression for diffusion transformer models, 2024

    Yuan, Z., Zhang, H., Lu, P., Ning, X., Zhang, L., Zhao, T., Yan, S., Dai, G., and Wang, Y. Ditfastattn: Attention compression for diffusion transformer models, 2024. URL https://arxiv.org/abs/2406.08552

    2024 arXiv

  52. [61]

    P., Jampani, V., Sun, D., and Yang, M.-H

    Zhang, J., Herrmann, C., Hur, J., Cabrera, L. P., Jampani, V., Sun, D., and Yang, M.-H. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. 2023 a

    2023

  53. [62]

    Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization, 2024

    Zhang, J., Huang, H., Zhang, P., Wei, J., Zhu, J., and Chen, J. Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization, 2024. URL https://arxiv.org/abs/2411.10958

    2024

  54. [63]

    Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration

    Zhang, J., Wei, J., Zhang, P., Zhu, J., and Chen, J. Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration. In International Conference on Learning Representations (ICLR), 2025 a

    2025

  55. [64]

    Spargeattn: Accurate sparse attention accelerating any model inference

    Zhang, J., Xiang, C., Huang, H., Wei, J., Xi, H., Zhu, J., and Chen, J. Spargeattn: Accurate sparse attention accelerating any model inference. arXiv preprint arXiv:2502.18137, 2025 b

    2025

  56. [65]

    A., Shechtman, E., and Wang, O

    Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018

    2018

  57. [66]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., R \'e , C., Barrett, C., et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36: 0 34661--34710, 2023 b

    2023

  58. [67]

    H _2 o: Heavy-hitter oracle for efficient generative inference of large language models, 2023 c

    Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., and Chen, B. H _2 o: Heavy-hitter oracle for efficient generative inference of large language models, 2023 c . URL https://arxiv.org/abs/2306.14048

    2023 arXiv

  59. [68]

    Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation

    Zhao, T., Fang, T., Liu, E., Rui, W., Soedarmadji, W., Li, S., Lin, Z., Dai, G., Yan, S., Yang, H., et al. Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation. arXiv preprint arXiv:2406.02540, 2024 a

    2024 arXiv

  60. [69]

    Real-time video generation with pyramid attention broadcast, 2024 b

    Zhao, X., Jin, X., Wang, K., and You, Y. Real-time video generation with pyramid attention broadcast, 2024 b . URL https://arxiv.org/abs/2408.12588

    2024 arXiv

  61. [70]

    Atom: Low-bit quantization for efficient and accurate llm serving

    Zhao, Y., Lin, C.-Y., Zhu, K., Ye, Z., Chen, L., Zheng, S., Ceze, L., Krishnamurthy, A., Chen, T., and Kasikci, B. Atom: Low-bit quantization for efficient and accurate llm serving. MLSys, 2024 c

    2024

  62. [71]

    Pit: Optimization of dynamic sparse deep learning models via permutation invariant transformation, 2023

    Zheng, N., Jiang, H., Zhang, Q., Han, Z., Yang, Y., Ma, L., Yang, F., Zhang, C., Qiu, L., Yang, M., and Zhou, L. Pit: Optimization of dynamic sparse deep learning models via permutation invariant transformation, 2023. URL https://arxiv.org/abs/2301.10936

    2023 arXiv

  63. [72]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.