Pith. sign in

REVIEW 5 major objections 3 minor 77 references

A single head-classification at maximum noise lets autoregressive video diffusion run up to 1.95× faster at 1080P, with no retraining and no flicker.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:40 UTC pith:SY6IBORW

load-bearing objection A practical training-free acceleration for AR video diffusion that mostly delivers; the one-shot stability assumption deserves a long-horizon stress test before taking the 30s speedups at face value. the 5 major comments →

arxiv 2607.20125 v1 pith:SY6IBORW submitted 2026-07-22 cs.CV cs.LG

HeadCast: Casting Attention Heads for Efficient Autoregressive Video Generation

Jinliang Shen , Lianghao Su , Zheming Li , Kang He , ZiLiang Lai , Yanbing Jiang , Chengru Song This is my paper
classification cs.CV cs.LG
keywords autoregressive video diffusionKV cache compressionattention head archetypestraining-free accelerationsparse attentiontemporal consistencydiffusion transformerflicker reduction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that attention heads in pre-trained autoregressive video diffusion models have stable, distinct personalities: Sink heads anchor on the first frame, Dummy heads read only the newest block, Spatial heads look at a local neighborhood across history, and only Global heads need the whole context. If true, a single classification — taken at the noisiest denoising step, where attention shows structure rather than content — is enough to route every head for the rest of generation. On that basis the paper builds HeadCast, a training-free plug-in that splits the ever-growing key-value cache into head-specific buffers and gives each head its own attention path. The fixed-grid Spatial path makes savings grow with resolution, reaching up to 1.62× at 720P and 1.95× at 1080P while VBench stays within 0.15 points of full attention and frame-level fidelity lands at 22–27 dB PSNR versus 16–19 dB for the aggressive-eviction baseline. A fair reader cares because attention over the KV cache dominates inference cost in long, high-resolution video generation — and this attack needs no re-training.

Core claim

On the paper's own terms, the discovery is that autoregressive video diffusion models do not need every attention head to see the full history: heads segregate into four stable archetypes defined by where their attention mass falls, and assignments survive across autoregressive steps (90.3% identical between frames 12 and 18), denoising timesteps (85.8%), and prompts (79.8%). HeadCast classifies each head once, at maximum noise (t=1000), by cosine similarity between its full-context output and three restricted proxies, then restructures the monolithic KV cache: Sink and Dummy heads keep a single block, Spatial heads attend within a fixed 10×10 grid across history, and only Global heads retai

What carries the argument

The carrying mechanism is a four-way head classification that feeds per-archetype cache restructuring. When the sliding window first fills, HeadCast compares each head's full-context output against three restricted proxies — attention over only the first block (Sink), only the current block (Dummy), and only a local neighborhood (Spatial) — and assigns the head to the first archetype whose cosine similarity to the reference clears 0.95, with a conservative 5th-percentile-plus-MSE score gating Spatial. The KV cache is then split along the head dimension: Sink and Dummy share a single-block buffer, Global keeps the full sliding window, and Spatial keeps full history in time but partitions each

Load-bearing premise

Heads assigned once from a single block at maximum noise keep their archetype for the entire rollout — yet stability is only measured between frame 12 and frame 18 (90.3% identical) and across prompts (79.8%), so a head that later drifts toward Global behavior would be permanently routed without the long-range keys it needs.

What would settle it

Re-run the four-way classification at every autoregressive step across a 30-second clip containing a scene transition, then compare output fidelity against the one-shot assignment: if per-step reclassification improves PSNR/LPIPS against full attention by more than about 1 dB, or if a single head tracked from Dummy or Sink to Global coincides with visible flicker at a block boundary, the one-time-classification premise fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • On a 30-second 1080P backbone, HeadCast's steady-state speedup reaches 1.95× (1.62× at 720P) while VBench Total stays within 0.15 points of full attention — the acceleration is nearly quality-neutral by the paper's absolute-quality metric.
  • Frame-level fidelity, the axis that catches flicker, lands at 22–27 dB PSNR versus 16–19 dB for Dummy Forcing on 5-second backbones (LPIPS 3–4× lower), and the block-boundary discontinuity metric matches full attention — implying the retained Global heads eliminate the pop-in artifacts coarse eviction causes.
  • The speedup grows with resolution and rollout length: the fixed-grid Spatial path touches a shrinking fraction of the growing cache, and the paper's FLOPs analysis gives roughly a 64% attention-FLOP reduction at 480P.
  • The one-time classification costs under 1.5% of total generation time on 30-second clips and amortizes to near zero on longer rollouts, so the method's relative advantage improves as videos get longer.
  • In blind side-by-side viewing, 90.9% of choices favor HeadCast over Dummy Forcing, while full attention still wins 60.4% of comparisons against HeadCast — a modest, often imperceptible quality cost for the speedup.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The stability figures cover one scene regime (frames 12–18) and vary with content (79.8% across prompts), so a cheap periodic re-classification at detected scene boundaries could recover the roughly 10–20% of heads that switch archetype — a natural streaming-mode extension the paper does not explore.
  • The deployed 10×10 grid is a coarser stand-in for the r=2 sliding-window neighborhood the classifier probes; per-head or per-resolution grid sizes could trade some speed for part of the ~1 dB PSNR that the paper's own threshold sweep shows is on the table.
  • The four-archetype structure is a claim about causal attention generally, so the same classify-once, split-the-cache recipe should transfer to other causal sequence models with growing KV caches, where stream-attention and retrieval-head phenomena have been observed separately but not unified into per-head cache pathways.
  • If archetype stability holds at minute scale, the classification overhead (already under 1.5% at 30 seconds) becomes negligible and the fixed-grid savings keep compounding, suggesting the method's ceiling is best measured on 2–3 minute videos with scene transitions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 3 minor

Summary. HeadCast is a training-free, plug-and-play inference acceleration method for autoregressive (AR) video diffusion models. It observes that attention heads in such models exhibit stable, heterogeneous patterns and classifies each head once, at the maximum-noise denoising step t=1000, into one of four archetypes: Sink, Dummy, Spatial, or Global. The classification is based on cosine similarity between full-context and restricted-context attention outputs. HeadCast then restructures the KV cache into head-specific pathways: Sink/Dummy heads keep only one historical block, Spatial heads attend within a fixed non-overlapping grid, and Global heads retain the full sliding window. The paper claims up to 1.62x speedup at 720P and 1.95x at 1080P on a 30-second LongLive backbone, with VBench quality on par with full attention and substantially higher PSNR/LPIPS than Dummy Forcing, which it identifies as the most comparable training-free baseline. The method is evaluated on Self-Forcing, LongLive, Causal Forcing, and Reward Forcing, with additional ablations, a BBD flicker metric, and a user study.

Significance. If the one-shot classification stability and the Spatial proxy-to-deployed-grid transfer hold, HeadCast is a practical, training-free speedup that can be plugged into several AR video backbones, with savings that grow with resolution. The paper's strengths are its breadth of backbones, the high-resolution scaling study, the BBD metric designed to expose block-boundary flicker, the ablation of archetype contributions, and the release of code. The head-archetype taxonomy itself is a useful analysis contribution. The main risk is the central stability premise: the evidence for one-shot classification is limited to short-range AR-step agreement, denoising-timestep agreement, and prompt agreement, with no long-horizon or scene-transition stress test. A secondary risk is the mismatch between the Spatial classification criterion and the deployed Spatial grid, which is acknowledged but not quantified. These concerns are addressable with additional experiments, so the central claim remains defensible.

major comments (5)
  1. [§3.2 (Observation 3), §5.5 (Stability of the One-Shot Classification), Fig. 6, Table 8] The one-time classification premise is load-bearing, but the stability evidence covers only frame 12 vs. frame 18 for the AR-step axis (90.3%), and 79.8% across prompts. The 30-second flagship speedups and the BBD results in Table 8 all use this one-shot assignment throughout the rollout. If a head that would become Global after a scene transition or long-horizon drift is permanently routed to Sink/Dummy/Spatial, it loses the long-range keys needed for temporal consistency—exactly the failure mode HeadCast claims to avoid. The paper never tests this with scene transitions or an oracle/periodic-reclassification comparison. Without such a stress test, the stability premise is under-supported for the 30s setting where the headline speedups are reported.
  2. [§4.2 (Eq. 3–5), §4.4, §5.1] The Spatial proxy used for classification is a sliding (2r+1)x(2r+1) neighborhood with r=2, while the deployed Spatial path partitions each frame into non-overlapping 10x10 cells. The paper states this difference is intentional for FlashAttention efficiency, but it never measures how often the two criteria agree. A head may have high cosine similarity under a sliding 5x5 window yet poor behavior when forced into a 10x10 cell (or vice versa), so the classification does not necessarily match the deployed computation. This mismatch is not ablated; it could shift both the measured speed and fidelity. Please report the agreement between proxy-based and grid-based routing, or use the same spatial neighborhood in classification and deployment.
  3. [§5.1, §5.5 (Impact of Classification Thresholds), Fig. 5, Table 6] The thresholds θ_s=θ_d=θ_sp=0.95, θ_sc=0.755, and the MSE weight γ=5 are tuned via ablation on Self-Forcing at 5s/720P and then applied unchanged to all backbones and resolutions. Table 6 shows that head distributions vary materially with resolution (e.g., Spatial 168.2→203.2 and Global 96.5→63.7 for LongLive), so threshold sensitivity is not negligible. The paper claims these thresholds are architecture-agnostic because they compare normalized cosine similarities, but no cross-backbone or cross-resolution sensitivity analysis is provided. A sensitivity sweep on at least one other backbone/resolution is needed to support the generality claim.
  4. [§5.1, Appendix D, Tables 1–2] All headline speedups are steady-state, excluding the one-time classification cost. For 5-second clips this overhead is 5–8% of total time (Table 7), so end-to-end speedups are materially lower: e.g., Self-Forcing 720P 1.31x becomes roughly 1.22x when the 7.1% overhead is included. The abstract and conclusion state 'up to 1.62x/1.95x' without this qualification, and the 5s numbers are presented as 'speedup' in Table 1. Although the 30s overhead is under 1.5%, the reporting is misleading for the 5s setting. Please report end-to-end speedups in the main tables or clearly qualify the steady-state basis in the abstract.
  5. [§5.1, Tables 1–2] VBench scores are reported as single numbers averaged over 5 seeds, without variance or significance testing. The central 'on par within 0.15 points' claim relies on differences that may be within seed noise, particularly for the 30s LongLive setting where only one backbone is used. Please provide confidence intervals or statistical tests, or at minimum report per-seed ranges, to support the VBench-parity claim.
minor comments (3)
  1. [Figure 2] The label 'UniformedFullAttention' appears garbled; it should likely read 'Uniform Full Attention'.
  2. [Table 2] The baseline PSNR is shown as '∞/—'; the legend explains this, but the notation is unconventional. Suggest using '—' only, since '∞' is not a PSNR value.
  3. [§5.3] The claim that lower PSNR on LongLive is 'a property of the reference-based metric, not a loss of quality' would be strengthened by quantitative evidence of semantic faithfulness (e.g., CLIP scores) rather than only qualitative frames.

Circularity Check

0 steps flagged

No significant circularity: central speed/quality claims are externally benchmarked; the stability check is a self-consistency measure but not a derivation.

full rationale

HeadCast is an empirical inference-acceleration method, not a derivation. The one-time head classification (Eqs. 2-5) is a heuristic routing rule whose thresholds are ablated on one model and then applied to others; the resulting FPS, PSNR/LPIPS, VBench, and BBD numbers are measured against full attention and Dummy Forcing, which are external references. The only self-referential element is the stability claim in Sec. 5.5: the 'identical archetype' fractions (90.3%, 85.8%, 79.8%) are agreement rates of the same classifier (Eq. 5) with itself, so Observation 3 is partly a reproducibility check. However, this is not a case where a predicted quantity is the fitted input: the downstream quality/speed results are independent measurements, and the paper does not use the stability percentages to compute its headline speedup or VBench parity. There are no load-bearing self-citations, no uniqueness theorems imported from the authors' prior work, and no renaming of known results as new predictions. The under-tested long-horizon drift (frame 12 vs 18 only, no scene-transition stress test) is a validity limitation, not circular reasoning.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

No new physical or architectural entities are introduced. The four archetypes are descriptive labels for observed behavior. The main load-bearing elements are fitted thresholds, the stability assumption, and the grid-vs-sliding-window approximation.

free parameters (6)
  • cosine threshold θ (θ_s=θ_d=θ_sp) = 0.95
    Gates Sink/Dummy/Spatial assignment; chosen via ablation on Self-Forcing (§5.5) and applied unchanged to all backbones.
  • spatial score threshold θ_sc = 0.755
    Gates Spatial assignment via worst-case score; tuned in ablation (§5.5).
  • MSE penalty weight γ = 5
    Weight in the spatial proxy score (Eq. 4); chosen by hand.
  • Spatial proxy radius r = 2
    Sliding (2r+1)×(2r+1) neighborhood used only for classification (§5.1); deployed path uses a different non-overlapping grid.
  • deployed spatial grid cell = 10x10 with rounded remainder
    Partitions each latent frame into non-overlapping cells for FlashAttention; chosen for hardware efficiency rather than fidelity (§4.4).
  • classification denoising timestep t = 1000
    One-shot classification performed at the maximum-noise step; justified by ablation (Table 5) but not derived.
axioms (4)
  • domain assumption Pre-trained AR video DiT attention heads exhibit stable, heterogeneous archetypes (Observations 1-3)
    The whole method depends on the empirical claim that heads can be typed once and remain stable; evidence is §3.2 and Figure 6.
  • domain assumption Attention output similarity under restricted contexts is a faithful proxy for routing safety
    HeadCast decides modes by cosine similarity and MSE between restricted and full attention outputs (Eqs. 2-4), assuming high similarity means safe to restrict.
  • ad hoc to paper Non-overlapping 10x10 grid approximates the sliding 5x5 neighborhood used in classification
    The deployed Spatial path uses a different spatial mask than the one used to classify Spatial heads (§4.4); no proof is given that grid cells preserve the same locality.
  • domain assumption VBench plus reference-based PSNR/LPIPS capture quality and flicker adequately
    Headline quality claims rest on VBench, which the paper itself shows is insensitive to flicker; PSNR/LPIPS are w.r.t. the full-attention output, not ground truth. A user study covers only 160 pairs.

pith-pipeline@v1.3.0-alltime-deepseek · 19876 in / 9803 out tokens · 93315 ms · 2026-08-01T10:40:50.665082+00:00 · methodology

0 comments
read the original abstract

Autoregressive (AR) video diffusion models have become a promising paradigm for long and streaming video synthesis, but the continuously growing Key-Value (KV) cache makes attention the dominant inference cost, especially at high resolution where each frame contributes many tokens. Existing remedies either evict the cache with coarse heuristics that cause inter-frame flickering, or require model re-training. We propose HeadCast, a training-free, plug-and-play acceleration framework built on the observation that a pre-trained AR model's attention heads exhibit stable, heterogeneous behaviors. After a short warm-up, HeadCast performs a one-time classification at the maximum-noise step that sorts every head into one of four archetypes: Sink, Dummy, Spatial, and Global, and restructures the monolithic KV cache into head-specific pathways. Crucially, it retains the Global heads that preserve the long-range temporal consistency aggressive eviction destroys. Because the Spatial pathway operates on a fixed-size grid, its savings grow with resolution: across state-of-the-art AR models, HeadCast accelerates inference by up to 1.62x at 720P and 1.95x at 1080P, while keeping VBench quality on par with full attention and largely flicker-free. Code is available at https://github.com/sjlgaga/HeadCast .

Figures

Figures reproduced from arXiv: 2607.20125 by Chengru Song, Jinliang Shen, Kang He, Lianghao Su, Yanbing Jiang, Zheming Li, Ziliang Lai.

Figure 1
Figure 1. Figure 1: The four attention-head archetypes in a pre-trained AR video model. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The four-phase HeadCast pipeline: full-context Warm-up, Online Classification, Heterogeneous Cache Management, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The Spatial Path. Each frame is partitioned into [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Inter-frame flickering at an AR block boundary. Across the block edge at frame 129, Dummy Forcing’s eviction pops in a spurious structure (a tree on the right, red box, frame 130), while Full Attention and HeadCast stay temporally consistent. et al. 2025); every VBench score is averaged over 5 seeds per prompt. As text-to-video generation has no ground-truth frames, we use each model’s own full-attention o… view at source ↗
Figure 5
Figure 5. Figure 5: Ablation of classification thresholds. Left: spa￾tial score threshold θsc. Right: cosine threshold θ. Lowering either threshold routes more heads to sparse paths, trad￾ing PSNR for FPS; green dashed lines mark our defaults (θsc=0.755, θ=0.95). As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Stability of the one-shot classification. Each bar is [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Sink, Dummy, and Spatial heads. One representative head per archetype, each occupying two rows (frame 6 then frame 18). Columns: raw map, per-frame (temporal) aggregation, per-position (spatial) aggregation. The Sink head (Layer 23, Head 2) concentrates on the first cached block; the Dummy head (Layer 28, Head 10) collapses onto the current block; the Spatial head (Layer 27, Head 1) attends to a fixed loca… view at source ↗
Figure 8
Figure 8. Figure 8: Global head. One representative head occupying two rows (frame 6 then frame 18). Columns: raw map, per-frame (temporal) aggregation, per-position (spatial) aggregation. The Global head (Layer 18, Head 9) spreads its attention broadly across the whole history. one—so the added kernel-launch overhead offsets part of the FLOPs reduction. C KV-Cache Memory Cost A KV-cache compression method should also shrink … view at source ↗
Figure 9
Figure 9. Figure 9: Steady-state KV-cache size (GB) for the full [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Additional inter-frame flickering examples at AR block boundaries (LongLive, 30 s, 720P). Each five-frame window is centered on a block edge. Dummy Forcing’s aggressive cache eviction introduces an abrupt, frame-to-frame change in the boxed region, whereas Full Attention and our HeadCast keep the scene temporally consistent. Within each pair the two videos are shown as video 1/video 2 in randomized order,… view at source ↗
Figure 11
Figure 11. Figure 11: Additional qualitative results on Self-Forcing 5s (720P), part 1. Rows top to bottom: Full Attention, HeadCast. Five frames are sampled uniformly across the 5-second clip. Frame 0 Frame 20 Frame 40 Frame 60 Frame 80 Full Attention HeadCast (a) A drone shot circling a historic church on the Amalfi Coast. Frame 0 Frame 20 Frame 40 Frame 60 Frame 80 Full Attention HeadCast (b) Golden retriever puppies playin… view at source ↗
Figure 12
Figure 12. Figure 12: Additional qualitative results on Self-Forcing 5s (720P), part 2. Rows top to bottom: Full Attention, HeadCast. Five frames are sampled uniformly across the 5-second clip. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Additional qualitative results on LongLive 30s (720P), part 1. Rows top to bottom: Full Attention, HeadCast. Five frames are sampled uniformly across the 30-second clip. Frame 0 Frame 125 Frame 250 Frame 375 Frame 500 Full Attention HeadCast (a) Red pandas in a bamboo terrarium. Frame 0 Frame 125 Frame 250 Frame 375 Frame 500 Full Attention HeadCast (b) An elderly man walking along a city street [PITH_FU… view at source ↗
Figure 14
Figure 14. Figure 14: Additional qualitative results on LongLive 30s (720P), part 2. Rows top to bottom: Full Attention, HeadCast. Five frames are sampled uniformly across the 30-second clip. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 11 linked inside Pith

  1. [1]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

    From Slow Bidirectional to Fast Autoregressive Video Diffusion Models , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

  2. [2]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  3. [3]

    Yang, Shuai and Huang, Wei and Chu, Ruihang and Xiao, Yicheng and Zhao, Yuyang and Wang, Xianbang and Li, Muyang and Xie, Enze and Chen, Yingcong and Lu, Yao and Han, Song and Chen, Yukang , booktitle =

  4. [4]

    International Conference on Learning Representations (ICLR) , year =

    Rolling Forcing: Autoregressive Long Video Diffusion in Real Time , author =. International Conference on Learning Representations (ICLR) , year =

  5. [5]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Diffusion Forcing: Next-Token Prediction Meets Full-Sequence Diffusion , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  6. [8]

    Proceedings of the International Conference on Machine Learning (ICML) , year =

    Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression , author =. Proceedings of the International Conference on Machine Learning (ICML) , year =

  7. [11]

    Lv, Chengtao and Shi, Yumeng and Huang, Yushi and Gong, Ruihao and Ren, Shen and Wang, Wenya , journal =

  8. [13]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

    Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

  9. [14]

    Proceedings of the International Conference on Machine Learning (ICML) , year =

    Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation , author =. Proceedings of the International Conference on Machine Learning (ICML) , year =

  10. [15]

    International Conference on Learning Representations (ICLR) , year =

    Efficient Streaming Language Models with Attention Sinks , author =. International Conference on Learning Representations (ICLR) , year =

  11. [16]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Zhang, Zhenyu and Sheng, Ying and Zhou, Tianyi and Chen, Tianlong and Zheng, Lianmin and Cai, Ruisi and Song, Zhao and Tian, Yuandong and R. Advances in Neural Information Processing Systems (NeurIPS) , year =

  12. [17]

    Li, Yuhong and Huang, Yingbing and Yang, Bowen and Venkitesh, Bharat and Locatelli, Acyr and Ye, Hanchen and Cai, Tianle and Lewis, Patrick and Chen, Deming , booktitle =

  13. [18]

    Xiao, Guangxuan and Tang, Jiaming and Zuo, Jingwei and Guo, Junxian and Yang, Shang and Tang, Haotian and Fu, Yao and Han, Song , booktitle =

  14. [19]

    Model Tells You What to Discard: Adaptive

    Ge, Suyu and Zhang, Yunan and Liu, Liyuan and Zhang, Minjia and Han, Jiawei and Gao, Jianfeng , booktitle =. Model Tells You What to Discard: Adaptive

  15. [20]

    Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , year =

    Scalable Diffusion Models with Transformers , author =. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , year =

  16. [21]

    Proceedings of the International Conference on Machine Learning (ICML) , year =

    Fast Video Generation with Sliding Tile Attention , author =. Proceedings of the International Conference on Machine Learning (ICML) , year =

  17. [22]

    Zhang, Peiyuan and Huang, Haofeng and Chen, Yongqi and Su, Runlong and Liu, Zhengzhong and Stoica, Ion and Xing, Eric and Zhang, Hao , booktitle =

  18. [23]

    Xi, Haocheng and Yang, Shuo and Zhao, Yilong and Xu, Chenfeng and Li, Muyang and Li, Xiuyu and Lin, Yujun and Cai, Han and Zhang, Jintao and Li, Dacheng and Chen, Jianfei and Stoica, Ion and Keutzer, Kurt and Han, Song , booktitle =. Sparse

  19. [24]

    Yang, Shuo and Xi, Haocheng and Zhao, Yilong and Li, Muyang and Zhang, Jintao and Cai, Han and Lin, Yujun and Li, Xiuyu and Xu, Chenfeng and Chen, Jianfei and Han, Song and Keutzer, Kurt and Stoica, Ion , booktitle =. Sparse

  20. [26]

    Yang, An and others , journal =

  21. [28]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Denoising Diffusion Probabilistic Models , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  22. [29]

    International Conference on Learning Representations (ICLR) , year =

    Denoising Diffusion Implicit Models , author =. International Conference on Learning Representations (ICLR) , year =

  23. [30]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

    High-Resolution Image Synthesis with Latent Diffusion Models , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

  24. [31]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Attention Is All You Need , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  25. [32]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Video Diffusion Models , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  26. [34]

    Yang, Zhuoyi and Teng, Jiayan and Zheng, Wendi and Ding, Ming and Huang, Shiyu and Xu, Jiazheng and Yang, Yuanming and Hong, Wenyi and Zhang, Xiaohan and Feng, Guanyu and others , booktitle =

  27. [35]

    Kong, Weijie and Tian, Qi and Zhang, Zijian and Min, Rox and Dai, Zuozhuo and Zhou, Jin and Xiong, Jiangfeng and Li, Xin and Wu, Bo and Zhang, Jianwei and others , journal =

  28. [37]

    International Conference on Learning Representations (ICLR) , year =

    Progressive Distillation for Fast Sampling of Diffusion Models , author =. International Conference on Learning Representations (ICLR) , year =

  29. [38]

    Proceedings of the International Conference on Machine Learning (ICML) , year =

    Consistency Models , author =. Proceedings of the International Conference on Machine Learning (ICML) , year =

  30. [39]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

    One-step Diffusion with Distribution Matching Distillation , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

  31. [40]

    and Ermon, Stefano and Rudra, Atri and R

    Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R. Advances in Neural Information Processing Systems (NeurIPS) , year =

  32. [41]

    Liu, Zirui and Yuan, Jiayi and Jin, Hongye and Zhong, Shaochen and Xu, Zhaozhuo and Braverman, Vladimir and Chen, Beidi and Hu, Xia , booktitle =

  33. [42]

    Huang, Ziqi and He, Yinan and Yu, Jiashuo and Zhang, Fan and Si, Chenyang and Jiang, Yuming and Zhang, Yuanhan and Wu, Tianxing and Jin, Qingyang and Chanpaisit, Nattapol and Wang, Yaohui and Chen, Xinyuan and Wang, Limin and Lin, Dahua and Qiao, Yu and Liu, Ziwei , booktitle =

  34. [43]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

    The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

  35. [44]

    Blattmann, A.; Dockhorn, T.; Kulal, S.; Mendelevitch, D.; Kilian, M.; Lorenz, D.; Levi, Y.; English, Z.; Voleti, V.; Letts, A.; et al. 2023. Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets. arXiv preprint arXiv:2311.15127

    Pith/arXiv arXiv 2023

  36. [45]

    M.; Du, Y.; Simchowitz, M.; Tedrake, R.; and Sitzmann, V

    Chen, B.; Mons \'o , D. M.; Du, Y.; Simchowitz, M.; Tedrake, R.; and Sitzmann, V. 2024. Diffusion Forcing: Next-Token Prediction Meets Full-Sequence Diffusion. In Advances in Neural Information Processing Systems (NeurIPS)

    2024

  37. [46]

    Cui, J.; Wu, J.; Li, M.; Yang, T.; Li, X.; Wang, R.; Bai, A.; Ban, Y.; and Hsieh, C.-J. 2025. Self-Forcing++: Towards Minute-Scale High-Quality Video Generation. arXiv preprint arXiv:2510.02283

    Pith/arXiv arXiv 2025

  38. [47]

    Y.; Ermon, S.; Rudra, A.; and R \'e , C

    Dao, T.; Fu, D. Y.; Ermon, S.; Rudra, A.; and R \'e , C. 2022. FlashAttention : Fast and Memory-Efficient Exact Attention with IO -Awareness. In Advances in Neural Information Processing Systems (NeurIPS)

    2022

  39. [48]

    Ge, S.; Zhang, Y.; Liu, L.; Zhang, M.; Han, J.; and Gao, J. 2024. Model Tells You What to Discard: Adaptive KV Cache Compression for LLM s. In International Conference on Learning Representations (ICLR)

    2024

  40. [49]

    Guo, H.; Jia, Z.; Li, J.; Li, B.; Cai, Y.; Wang, J.; Li, Y.; and Lu, Y. 2026. Efficient Autoregressive Video Diffusion with Dummy Head. arXiv preprint arXiv:2601.20499

    arXiv 2026

  41. [50]

    Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems (NeurIPS)

    2020

  42. [51]

    Ho, J.; Salimans, T.; Gritsenko, A.; Chan, W.; Norouzi, M.; and Fleet, D. J. 2022. Video Diffusion Models. In Advances in Neural Information Processing Systems (NeurIPS)

    2022

  43. [52]

    Huang, X.; Li, Z.; He, G.; Zhou, M.; and Shechtman, E. 2025. Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion. In Advances in Neural Information Processing Systems (NeurIPS)

    2025

  44. [53]

    Huang, Z.; He, Y.; Yu, J.; Zhang, F.; Si, C.; Jiang, Y.; Zhang, Y.; Wu, T.; Jin, Q.; Chanpaisit, N.; Wang, Y.; Chen, X.; Wang, L.; Lin, D.; Qiao, Y.; and Liu, Z. 2024. VBench : Comprehensive Benchmark Suite for Video Generative Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2024

  45. [54]

    Ji, Y.; et al. 2026. Forcing-KV: Hybrid KV Cache Compression for Efficient Autoregressive Video Diffusion Models. arXiv preprint arXiv:2605.09681

    Pith/arXiv arXiv 2026

  46. [55]

    Kong, W.; Tian, Q.; Zhang, Z.; Min, R.; Dai, Z.; Zhou, J.; Xiong, J.; Li, X.; Wu, B.; Zhang, J.; et al. 2024. HunyuanVideo : A Systematic Framework For Large Video Generative Models. arXiv preprint arXiv:2412.03603

    Pith/arXiv arXiv 2024

  47. [56]

    Li, Y.; Huang, Y.; Yang, B.; Venkitesh, B.; Locatelli, A.; Ye, H.; Cai, T.; Lewis, P.; and Chen, D. 2024. SnapKV : LLM Knows What You Are Looking for Before Generation. In Advances in Neural Information Processing Systems (NeurIPS)

    2024

  48. [57]

    Liu, K.; Hu, W.; Xu, J.; Shan, Y.; and Lu, S. 2026. Rolling Forcing: Autoregressive Long Video Diffusion in Real Time. In International Conference on Learning Representations (ICLR)

    2026

  49. [58]

    Liu, Z.; Yuan, J.; Jin, H.; Zhong, S.; Xu, Z.; Braverman, V.; Chen, B.; and Hu, X. 2024. KIVI : A Tuning-Free Asymmetric 2bit Quantization for KV Cache. In Proceedings of the International Conference on Machine Learning (ICML)

    2024

  50. [59]

    L.; Zhu, J.; Cao, H.; Zhang, Z.; Zhu, X.; Shen, Y.; and Zhang, M

    Lu, Y.; Zeng, Y.; Li, H.; Ouyang, H.; Wang, Q.; Cheng, K. L.; Zhu, J.; Cao, H.; Zhang, Z.; Zhu, X.; Shen, Y.; and Zhang, M. 2026. Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2026

  51. [60]

    Lv, C.; Shi, Y.; Huang, Y.; Gong, R.; Ren, S.; and Wang, W. 2026. Light Forcing : Accelerating Autoregressive Video Diffusion via Sparse Attention. arXiv preprint arXiv:2602.04789

    Pith/arXiv arXiv 2026

  52. [61]

    Peebles, W.; and Xie, S. 2023. Scalable Diffusion Models with Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    2023

  53. [62]

    Polyak, A.; et al. 2024. Movie Gen: A Cast of Media Foundation Models. arXiv preprint arXiv:2410.13720

    Pith/arXiv arXiv 2024

  54. [63]

    Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2022

  55. [64]

    Salimans, T.; and Ho, J. 2022. Progressive Distillation for Fast Sampling of Diffusion Models. In International Conference on Learning Representations (ICLR)

    2022

  56. [65]

    Sand AI . 2025. MAGI-1 : Autoregressive Video Generation at Scale. arXiv preprint arXiv:2505.13211

    Pith/arXiv arXiv 2025

  57. [66]

    Song, J.; Meng, C.; and Ermon, S. 2021. Denoising Diffusion Implicit Models. In International Conference on Learning Representations (ICLR)

    2021

  58. [67]

    Song, Y.; Dhariwal, P.; Chen, M.; and Sutskever, I. 2023. Consistency Models. In Proceedings of the International Conference on Machine Learning (ICML)

    2023

  59. [68]

    Tian, J.; Wang, Y.; Yu, G.; and Zhang, C. 2026. Head Forcing: Long Autoregressive Video Generation via Head Heterogeneity. arXiv preprint arXiv:2605.14487

    Pith/arXiv arXiv 2026

  60. [69]

    N.; Kaiser, .; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems (NeurIPS)

    2017

  61. [70]

    Wan Team . 2025. Wan : Open and Advanced Large-Scale Video Generative Models. arXiv preprint arXiv:2503.20314

    Pith/arXiv arXiv 2025

  62. [71]

    Xi, H.; Yang, S.; Zhao, Y.; Xu, C.; Li, M.; Li, X.; Lin, Y.; Cai, H.; Zhang, J.; Li, D.; Chen, J.; Stoica, I.; Keutzer, K.; and Han, S. 2025. Sparse VideoGen : Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity. In Proceedings of the International Conference on Machine Learning (ICML)

    2025

  63. [72]

    Xiao, G.; Tang, J.; Zuo, J.; Guo, J.; Yang, S.; Tang, H.; Fu, Y.; and Han, S. 2025. DuoAttention : Efficient Long-Context LLM Inference with Retrieval and Streaming Heads. In International Conference on Learning Representations (ICLR)

    2025

  64. [73]

    Xiao, G.; Tian, Y.; Chen, B.; Han, S.; and Lewis, M. 2024. Efficient Streaming Language Models with Attention Sinks. In International Conference on Learning Representations (ICLR)

    2024

  65. [74]

    Xu, B.; Du, Y.; Liu, Z.; Yang, S.; Jiang, Z.; Yan, S.; Saha, R.; Pumarola, A.; Wang, W.; and Li, P. 2026. Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation. arXiv preprint arXiv:2604.21221

    Pith/arXiv arXiv 2026

  66. [75]

    Yang, A.; et al. 2024. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115

    Pith/arXiv arXiv 2024

  67. [76]

    Yang, S.; Huang, W.; Chu, R.; Xiao, Y.; Zhao, Y.; Wang, X.; Li, M.; Xie, E.; Chen, Y.; Lu, Y.; Han, S.; and Chen, Y. 2026. LongLive : Real-Time Interactive Long Video Generation. In International Conference on Learning Representations (ICLR)

    2026

  68. [77]

    Yang, S.; Xi, H.; Zhao, Y.; Li, M.; Zhang, J.; Cai, H.; Lin, Y.; Li, X.; Xu, C.; Chen, J.; Han, S.; Keutzer, K.; and Stoica, I. 2025 a . Sparse VideoGen2 : Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation. In Advances in Neural Information Processing Systems (NeurIPS)

    2025

  69. [78]

    Yang, Z.; Teng, J.; Zheng, W.; Ding, M.; Huang, S.; Xu, J.; Yang, Y.; Hong, W.; Zhang, X.; Feng, G.; et al. 2025 b . CogVideoX : Text-to-Video Diffusion Models with An Expert Transformer. In International Conference on Learning Representations (ICLR)

    2025

  70. [79]

    H.; Nam, J.; Yoon, H.; and Kim, S

    Yi, J.; Jang, W.; Cho, P. H.; Nam, J.; Yoon, H.; and Kim, S. 2026. Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression. In Proceedings of the International Conference on Machine Learning (ICML)

    2026

  71. [80]

    T.; and Park, T

    Yin, T.; Gharbi, M.; Zhang, R.; Shechtman, E.; Durand, F.; Freeman, W. T.; and Park, T. 2024. One-step Diffusion with Distribution Matching Distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2024

  72. [81]

    T.; Durand, F.; Shechtman, E.; and Huang, X

    Yin, T.; Zhang, Q.; Zhang, R.; Freeman, W. T.; Durand, F.; Shechtman, E.; and Huang, X. 2025. From Slow Bidirectional to Fast Autoregressive Video Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2025

  73. [82]

    Zhang, P.; Chen, Y.; Su, R.; Ding, H.; Stoica, I.; Liu, Z.; and Zhang, H. 2025 a . Fast Video Generation with Sliding Tile Attention. In Proceedings of the International Conference on Machine Learning (ICML)

    2025

  74. [83]

    Zhang, P.; Huang, H.; Chen, Y.; Su, R.; Liu, Z.; Stoica, I.; Xing, E.; and Zhang, H. 2025 b . VSA : Faster Video Diffusion with Trainable Sparse Attention. In Advances in Neural Information Processing Systems (NeurIPS)

    2025

  75. [84]

    A.; Shechtman, E.; and Wang, O

    Zhang, R.; Isola, P.; Efros, A. A.; Shechtman, E.; and Wang, O. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2018

  76. [85]

    Zhang, Z.; Sheng, Y.; Zhou, T.; Chen, T.; Zheng, L.; Cai, R.; Song, Z.; Tian, Y.; R \'e , C.; Barrett, C.; Wang, Z.; and Chen, B. 2023. H2O : Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS)

    2023

  77. [86]

    Zhu, H.; Zhao, M.; He, G.; Su, H.; Li, C.; and Zhu, J. 2026. Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation. In Proceedings of the International Conference on Machine Learning (ICML)

    2026