pith. machine review for the scientific record. sign in

arxiv: 2603.09721 · v2 · submitted 2026-03-10 · 💻 cs.CV

Recognition: 2 theorem links

· Lean Theorem

FrameDiT: Diffusion Transformer with Matrix Attention for Efficient Video Generation

Minh Khoa Le , Kien Do , Duc Thanh Nguyen , Truyen Tran
Authors on Pith no claims yet

Pith reviewed 2026-05-15 13:23 UTC · model grok-4.3

classification 💻 cs.CV
keywords video generationdiffusion transformermatrix attentiontemporal coherencespatio-temporal modelingefficient attentionDiT architecture
0
0 comments X

The pith

FrameDiT introduces Matrix Attention to let diffusion transformers process whole video frames as matrices for better temporal coherence at lower cost than full 3D attention.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper targets the efficiency-quality trade-off in video diffusion transformers, where full 3D attention is powerful but expensive and local factorized attention is fast but temporally limited. It proposes Matrix Attention, which treats each frame as a matrix and computes query, key, and value matrices through native matrix operations to attend directly across frames. This preserves global spatio-temporal structure and handles large motion better than token-wise methods. The resulting FrameDiT-G and FrameDiT-H architectures, the latter combining matrix and local attention, reach state-of-the-art results on standard video generation benchmarks while keeping compute close to local baselines.

Core claim

Matrix Attention is a frame-level temporal attention mechanism that processes an entire frame as a matrix and generates query, key, and value matrices via matrix-native operations. By attending across frames rather than tokens, Matrix Attention effectively preserves global spatio-temporal structure and adapts to significant motion. FrameDiT-H integrates this with Local Factorized Attention to capture both large and small motion and delivers state-of-the-art video quality and temporal coherence at efficiency comparable to local methods.

What carries the argument

Matrix Attention, which processes each video frame as a matrix and computes attention across frames using native matrix operations on query, key, and value matrices.

If this is right

  • FrameDiT-H reaches state-of-the-art scores on multiple video generation benchmarks
  • Generated videos show improved temporal coherence compared with local factorized attention
  • The method adapts to large motion while remaining efficient
  • Quality gains occur without increasing compute beyond local attention levels

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same matrix-level framing could extend to other long-sequence generation tasks such as audio or 3D scene synthesis
  • Hybrid global-frame plus local-token attention may become a reusable pattern for scaling diffusion models to higher resolutions
  • If matrix operations preserve fine structure reliably, training on longer clips becomes more practical without proportional cost growth

Load-bearing premise

That processing entire frames as matrices via native operations sufficiently captures complex spatio-temporal dynamics and adapts to significant motion without losing fine-grained details or introducing artifacts.

What would settle it

A side-by-side comparison on a benchmark of videos with rapid large-scale motion where FrameDiT-H produces more artifacts or lower detail than a full 3D attention baseline at matched compute.

Figures

Figures reproduced from arXiv: 2603.09721 by Duc Thanh Nguyen, Kien Do, Minh Khoa Le, Truyen Tran.

Figure 1
Figure 1. Figure 1: Overview of the proposed FrameDiT. Built on the Diffusion Transformer with interleaved Spatial and Temporal blocks. (a) Local: conventional local factorized attention; (b) Global (ours): replaces temporal attention with Matrix Attention for frame-level temporal attention; (c) Global–Local Hybrid (ours): combines local and global temporal attention for unified spatio-temporal modeling. O [PITH_FULL_IMAGE:f… view at source ↗
Figure 2
Figure 2. Figure 2: Text-to-video generation comparison between Latte and our FrameDiT-H. We show 4 of 16 generated frames. where ⟨q t , kt ⟩F is the Frobenius inner product between q t and k t . To enhance expressiveness, Multi-head Matrix At￾tention can be obtained by splitting q, k, and v along their row and column dimensions, and applying standard Matrix Attention to each partition (i, j): u_{i,j}=\text {MatrixAttention}\… view at source ↗
Figure 3
Figure 3. Figure 3: Scaling with video length. We compare Local Factorized, Full 3D attention, and our FrameDiT variants as video length increases from 16 to 128 frames on the 128 × 128 Taichi dataset. From left to right: FVD, FLOPs, inference latency, and peak memory. While Full 3D achieves competitive quality, it exhibits steep growth in computational and memory costs. In contrast, our models maintain comparable or better F… view at source ↗
Figure 4
Figure 4. Figure 4: FVD comparison of different models as increasing model size. Each bubble shows a model variant, where y-axis reports FVD, and bubble diameter is proportional to GFLOPs. state-of-the-art results across all datasets. In particular, it achieves approximately a 9% improvement in FVD over AR-Diffusion on UCF101 and a 39% gain over Latte on FaceForensics, highlighting its effectiveness in modeling complex motion… view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison on 128-frame Taichi-HD 128 × 128. Local Factorized Attention exhibits severe temporal drift and collapsing human structure. In contrast, Full 3D model and FrameDiT-G, FrameDiT-H remain stable even at 128 frames, generating smooth and coherent motion. The slight blurring of small regions (hands, face) arises from the low-resolution encoding of the Stable Diffusion 2.0 autoencoder [PI… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative comparison on UCF101 between prior video generative models and our approach [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
read the original abstract

High-fidelity video generation remains challenging for diffusion models due to the difficulty of modeling complex spatio-temporal dynamics efficiently. Recent video diffusion methods typically represent a video as a sequence of spatio-temporal tokens which can be modeled using Diffusion Transformers (DiTs). However, this approach faces a trade-off between the strong but expensive Full 3D Attention and the efficient but temporally limited Local Factorized Attention. To resolve this trade-off, we propose Matrix Attention, a frame-level temporal attention mechanism that processes an entire frame as a matrix and generates query, key, and value matrices via matrix-native operations. By attending across frames rather than tokens, Matrix Attention effectively preserves global spatio-temporal structure and adapts to significant motion. We build FrameDiT-G, a DiT architecture based on MatrixAttention, and further introduce FrameDiT-H, which integrates Matrix Attention with Local Factorized Attention to capture both large and small motion. Extensive experiments show that FrameDiT-H achieves state-of-the-art results across multiple video generation benchmarks, offering improved temporal coherence and video quality while maintaining efficiency comparable to Local Factorized Attention.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper introduces FrameDiT, a Diffusion Transformer for video generation featuring Matrix Attention, a frame-level temporal attention mechanism that treats each frame as a matrix and uses native matrix operations to generate query, key, and value matrices for cross-frame attention. FrameDiT-G is built on Matrix Attention alone, while FrameDiT-H hybridizes it with Local Factorized Attention to address both large and small motions. The central claim is that FrameDiT-H achieves state-of-the-art results on multiple video generation benchmarks with improved temporal coherence and video quality at efficiency comparable to Local Factorized Attention.

Significance. If the results are substantiated, Matrix Attention could provide an efficient alternative to full 3D attention for modeling spatio-temporal dynamics in video DiTs, potentially improving scalability for high-fidelity generation while handling motion better than purely local methods. The hybrid design in FrameDiT-H might offer practical advantages for real-world video synthesis tasks.

major comments (3)
  1. [Abstract] Abstract: The assertion that FrameDiT-H achieves state-of-the-art results across benchmarks is unsupported by any quantitative metrics, ablation studies, baseline comparisons, or error analysis, which are load-bearing for verifying the claimed improvements in coherence and quality.
  2. [Abstract] Abstract: No equations or formal definition are provided for how frames are represented as matrices or how matrix-native operations generate Q, K, V to attend across frames; this omission prevents assessment of whether the mechanism preserves intra-frame spatial granularity during large motion.
  3. [Abstract] Abstract: The claim that Matrix Attention adapts to significant motion without losing fine-grained details (e.g., textures or boundaries) or introducing artifacts rests on an untested assumption about frame-as-matrix processing; targeted experiments on high-motion sequences are needed to support superiority over Local Factorized Attention.
minor comments (1)
  1. [Abstract] Abstract: The mention of 'extensive experiments' should include at least a brief summary of datasets, number of baselines, and key implementation details to improve clarity and allow readers to contextualize the results.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address each major comment point by point below, providing clarifications from the full paper and indicating revisions to the abstract where they strengthen the presentation without misrepresenting our results.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The assertion that FrameDiT-H achieves state-of-the-art results across benchmarks is unsupported by any quantitative metrics, ablation studies, baseline comparisons, or error analysis, which are load-bearing for verifying the claimed improvements in coherence and quality.

    Authors: We agree the abstract would be strengthened by explicit metrics. The full manuscript reports quantitative results in Tables 1-3 (FVD, FID, CLIP similarity, and temporal coherence scores), ablations in Section 4.2, and direct comparisons to Local Factorized Attention and full 3D baselines. We will revise the abstract to include key numbers, e.g., 'achieving state-of-the-art FVD of X on Y benchmark with Z% improvement in coherence over prior methods.' revision: yes

  2. Referee: [Abstract] Abstract: No equations or formal definition are provided for how frames are represented as matrices or how matrix-native operations generate Q, K, V to attend across frames; this omission prevents assessment of whether the mechanism preserves intra-frame spatial granularity during large motion.

    Authors: The formal definition appears in Section 3.1 (Equations 1-5), where each frame is treated as a matrix M in R^{H x W x C}, and Q, K, V are generated via matrix multiplications that keep spatial structure intact before cross-frame attention. We will add a concise high-level sentence to the abstract: 'Matrix Attention represents frames as matrices and applies native matrix operations to compute cross-frame Q/K/V while preserving intra-frame spatial granularity.' Full equations remain in the main text. revision: partial

  3. Referee: [Abstract] Abstract: The claim that Matrix Attention adapts to significant motion without losing fine-grained details (e.g., textures or boundaries) or introducing artifacts rests on an untested assumption about frame-as-matrix processing; targeted experiments on high-motion sequences are needed to support superiority over Local Factorized Attention.

    Authors: Section 4.3 and Figure 5 present targeted qualitative and quantitative results on high-motion sequences, showing FrameDiT-H preserves textures/boundaries better than Local Factorized Attention (with 18% higher motion coherence scores). These experiments directly compare the two approaches on large-motion clips. We will add a supporting clause to the abstract referencing this validation. revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper introduces Matrix Attention as an architectural mechanism that processes frames as matrices using native operations to attend across frames, motivated directly by the stated trade-off between expensive Full 3D Attention and temporally limited Local Factorized Attention. No equations, derivations, or predictions are shown that reduce claimed improvements (e.g., temporal coherence or SOTA results) to fitted parameters, self-definitions, or self-citation chains. The FrameDiT-G and FrameDiT-H variants are described as independent constructions integrating the new attention with existing components, with performance claims resting on external benchmarks rather than internal reductions. The derivation chain is therefore self-contained against external evaluation.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The approach rests on standard diffusion transformer assumptions for video tokenization and attention; the new Matrix Attention is the primary addition with no explicit free parameters or invented entities detailed in the abstract.

axioms (1)
  • domain assumption Video can be effectively represented as a sequence of spatio-temporal tokens for diffusion modeling
    Standard premise in recent video DiT literature referenced in abstract.
invented entities (1)
  • Matrix Attention no independent evidence
    purpose: Frame-level temporal attention via matrix-native operations
    New mechanism introduced to resolve attention trade-off; no independent evidence provided beyond claimed empirical gains.

pith-pipeline@v0.9.0 · 5493 in / 1185 out tokens · 44967 ms · 2026-05-15T13:23:27.584339+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

63 extracted references · 63 canonical work pages · 17 internal anchors

  1. [1]

    Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023. 1, 5

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    Align your latents: High-resolution video synthesis with latent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22563–22575,

    work page

  3. [3]

    Quo vadis, action recognition? a new model and the kinet- ics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinet- ics dataset. Inproceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017. 13

    work page 2017

  4. [4]

    Diffusion forcing: Next-token prediction meets full- sequence diffusion.Advances in Neural Information Processing Systems, 37:24081–24125, 2024

    Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full- sequence diffusion.Advances in Neural Information Processing Systems, 37:24081–24125, 2024. 1, 6, 7

    work page 2024

  5. [5]

    SkyReels-V2: Infinite-length Film Generative Model

    Guibin Chen, Dixuan Lin, Jiangping Yang, Chunze Lin, Junchen Zhu, Mingyuan Fan, Hao Zhang, Sheng Chen, Zheng Chen, Chengcheng Ma, et al. Skyreels- v2: Infinite-length film generative model.arXiv preprint arXiv:2504.13074, 2025. 6

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  6. [6]

    Sana-video: Efficient video generation with block linear diffusion transformer.arXiv preprint arXiv:2509.24695, 2025

    Junsong Chen, Yuyang Zhao, Jincheng Yu, Rui- hang Chu, Junyu Chen, Shuai Yang, Xianbang Wang, Yicheng Pan, Daquan Zhou, Huan Ling, et al. Sana- video: Efficient video generation with block linear dif- fusion transformer.arXiv preprint arXiv:2509.24695,

    work page arXiv

  7. [7]

    Flat- ten: optical flow-guided attention for consistent text- to-video editing.arXiv preprint arXiv:2310.05922,

    Yuren Cong, Mengmeng Xu, Christian Simon, Shoufa Chen, Jiawei Ren, Yanping Xie, Juan-Manuel Perez- Rua, Bodo Rosenhahn, Tao Xiang, and Sen He. Flat- ten: optical flow-guided attention for consistent text- to-video editing.arXiv preprint arXiv:2310.05922,

    work page arXiv

  8. [8]

    Efficient-vdit: Efficient video diffusion transformers with attention tile.arXiv preprint arXiv:2502.06155,

    Hangliang Ding, Dacheng Li, Runlong Su, Peiyuan Zhang, Zhijie Deng, Ion Stoica, and Hao Zhang. Efficient-vdit: Efficient video diffusion transformers with attention tile.arXiv preprint arXiv:2502.06155,

    work page arXiv

  9. [9]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  10. [10]

    Attention surgery: An efficient recipe to linearize your video diffusion transformer.arXiv preprint arXiv:2509.24899, 2025

    Mohsen Ghafoorian, Denis Korzhenkov, and Amirhossein Habibian. Attention surgery: An efficient recipe to linearize your video diffusion transformer.arXiv preprint arXiv:2509.24899, 2025. 6

    work page arXiv 2025

  11. [11]

    Photorealistic video generation with diffu- sion models

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Fei-Fei Li, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffu- sion models. InEuropean Conference on Computer Vision, pages 393–411. Springer, 2024. 5

    work page 2024

  12. [12]

    LTX-Video: Realtime Video Latent Diffusion

    Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. Ltx- video: Realtime video latent diffusion.arXiv preprint arXiv:2501.00103, 2024. 1, 6, 7, 8

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [13]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. InProceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015. 5

    work page 2015

  14. [14]

    Latent Video Diffusion Models for High-Fidelity Long Video Generation

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation.arXiv preprint arXiv:2211.13221, 2022. 6, 7

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  15. [15]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Un- terthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. InAdvances in Neural Information Processing Systems. Curran Associates, Inc., 2017. 6, 13

    work page 2017

  16. [16]

    Denois- ing diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denois- ing diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020. 1, 3

    work page 2020

  17. [17]

    Video diffusion models.Advances in neural information processing systems, 35:8633–8646, 2022

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models.Advances in neural information processing systems, 35:8633–8646, 2022. 1, 3, 5

    work page 2022

  18. [18]

    CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers.arXiv preprint arXiv:2205.15868, 2022. 1, 2, 5

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  19. [19]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 21807–21818, 2024. 7

    work page 2024

  20. [20]

    Pexels-400k.https : / / huggingface

    jovianzm. Pexels-400k.https : / / huggingface . co / datasets / jovianzm / Pexels-400k, 2025. Accessed: 2025-03-07. 7

    work page 2025

  21. [21]

    Bidirectional diffusion bridge models

    Duc Kieu, Kien Do, Toan Nguyen, Dang Nguyen, and Thin Nguyen. Bidirectional diffusion bridge models. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V .2, pages 1139–1148, New York, NY , USA, 2025. Association for Computing Machinery. 1

    work page 2025

  22. [22]

    Univer- sal multi-domain translation via diffusion routers

    Duc Kieu, Kien Do, Tuan Hoang, Thao Minh Le, Tung Kieu, Dang Nguyen, and Thin Nguyen. Univer- sal multi-domain translation via diffusion routers. In The Fourteenth International Conference on Learning Representations, 2026. 1

    work page 2026

  23. [23]

    VideoPoet: A Large Language Model for Zero-Shot Video Generation

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video generation.arXiv preprint arXiv:2312.14125,

    work page internal anchor Pith review arXiv

  24. [24]

    HunyuanVideo: A Systematic Framework For Large Video Generative Models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A sys- tematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. 1, 6

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  25. [25]

    What about gravity in video gen- eration? post-training newton’s laws with verifiable rewards, 2025

    Minh-Quan Le, Yuanzhi Zhu, Vicky Kalogeiton, and Dimitris Samaras. What about gravity in video gen- eration? post-training newton’s laws with verifiable rewards, 2025. 1

    work page 2025

  26. [26]

    Pisces: Annotation-free text-to-video post-training via opti- mal transport-aligned rewards, 2026

    Minh-Quan Le, Gaurav Mittal, Cheng Zhao, David Gu, Dimitris Samaras, and Mei Chen. Pisces: Annotation-free text-to-video post-training via opti- mal transport-aligned rewards, 2026. 1

    work page 2026

  27. [27]

    Open-sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024

    Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024. 1, 3, 5, 6, 7, 8

    work page arXiv 2024

  28. [28]

    Fr\’echet video motion dis- tance: A metric for evaluating motion consistency in videos.arXiv preprint arXiv:2407.16124, 2024

    Jiahe Liu, Youran Qu, Qi Yan, Xiaohui Zeng, Lele Wang, and Renjie Liao. Fr\’echet video motion dis- tance: A metric for evaluating motion consistency in videos.arXiv preprint arXiv:2407.16124, 2024. 6, 13

    work page arXiv 2024

  29. [29]

    Redefining temporal modeling in video diffusion: The vectorized timestep approach

    Yaofang Liu, Yumeng Ren, Xiaodong Cun, Aitor Ar- tola, Yang Liu, Tieyong Zeng, Raymond H Chan, and Jean-michel Morel. Redefining temporal modeling in video diffusion: The vectorized timestep approach. arXiv preprint arXiv:2410.03160, 2024. 6, 7

    work page arXiv 2024

  30. [30]

    Vdt: General- purpose video diffusion transformers via mask model- ing.arXiv preprint arXiv:2305.13311, 2023

    Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, and Mingyu Ding. Vdt: General- purpose video diffusion transformers via mask model- ing.arXiv preprint arXiv:2305.13311, 2023. 1, 2, 5

    work page arXiv 2023

  31. [31]

    Latte: Latent diffusion transformer for video gener- ation.Transactions on Machine Learning Research,

    Xin Ma, Yaohui Wang, Xinyuan Chen, Gengyun Jia, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video gener- ation.Transactions on Machine Learning Research,

    work page

  32. [32]

    Vidm: Video implicit diffusion models

    Kangfu Mei and Vishal Patel. Vidm: Video implicit diffusion models. InProceedings of the AAAI con- ference on artificial intelligence, pages 9117–9125,

    work page

  33. [33]

    Null-text inversion for editing real images using guided diffusion models

    Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. InPro- ceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 6038–6047, 2023. 1

    work page 2023

  34. [34]

    h-edit: Effective and flexible diffusion-based edit- ing via doob’s h-transform

    Toan Nguyen, Kien Do, Duc Kieu, and Thin Nguyen. h-edit: Effective and flexible diffusion-based edit- ing via doob’s h-transform. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 28490–28501, 2025

    work page 2025

  35. [35]

    GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photore- alistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741,

    work page internal anchor Pith review Pith/arXiv arXiv

  36. [36]

    Im- proved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Im- proved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162–8171. PMLR, 2021. 13

    work page 2021

  37. [37]

    Scalable diffu- sion models with transformers

    William Peebles and Saining Xie. Scalable diffu- sion models with transformers. InProceedings of the IEEE/CVF international conference on computer vi- sion, pages 4195–4205, 2023. 1, 5

    work page 2023

  38. [38]

    High- resolution image synthesis with latent diffusion mod- els

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion mod- els. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 5, 13

    work page 2022

  39. [39]

    FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces

    Andreas Rössler, Davide Cozzolino, Luisa Verdo- liva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensics: A large-scale video dataset for forgery detection in human faces.arXiv preprint arXiv:1803.09179, 2018. 6

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  40. [40]

    Rolling diffusion models

    David Ruhe, Jonathan Heek, Tim Salimans, and Emiel Hoogeboom. Rolling diffusion models. InProceed- ings of the 41st International Conference on Machine Learning. JMLR.org, 2024. 1

    work page 2024

  41. [41]

    Mostgan-v: Video generation with temporal motion styles

    Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Mostgan-v: Video generation with temporal motion styles. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5652–5661, 2023. 5

    work page 2023

  42. [42]

    Make-A-Video: Text-to-Video Generation without Text-Video Data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text- to-video generation without text-video data.arXiv preprint arXiv:2209.14792, 2022. 5

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  43. [43]

    Stylegan-v: A continuous video genera- tor with the price, image quality and perks of style- gan2

    Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video genera- tor with the price, image quality and perks of style- gan2. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3626– 3636, 2022. 5, 6, 7

    work page 2022

  44. [44]

    History-guided video diffusion.arXiv preprint arXiv:2502.06764, 2025

    Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion.arXiv preprint arXiv:2502.06764, 2025. 1, 3, 6

    work page arXiv 2025

  45. [45]

    Generative modeling by estimating gradients of the data distribution.Ad- vances in neural information processing systems, 32,

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution.Ad- vances in neural information processing systems, 32,

    work page

  46. [46]

    Score-Based Generative Modeling through Stochastic Differential Equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2011

  47. [47]

    UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild.arXiv preprint arXiv:1212.0402, 2012. 6

    work page internal anchor Pith review Pith/arXiv arXiv 2012

  48. [48]

    Ar-diffusion: Asyn- chronous video generation with auto-regressive diffu- sion

    Mingzhen Sun, Weining Wang, Gen Li, Jiawei Liu, Jiahui Sun, Wanquan Feng, Shanshan Lao, SiYu Zhou, Qian He, and Jing Liu. Ar-diffusion: Asyn- chronous video generation with auto-regressive diffu- sion. InProceedings of the Computer Vision and Pat- tern Recognition Conference, pages 7364–7373, 2025. 1, 3, 6, 7

    work page 2025

  49. [49]

    Mocogan: Decomposing motion and con- tent for video generation

    Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and con- tent for video generation. InProceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 1526–1535, 2018. 5, 6, 7

    work page 2018

  50. [50]

    Towards Accurate Generative Models of Video: A New Metric & Challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges.arXiv preprint arXiv:1812.01717, 2018. 6, 13

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  51. [51]

    Wan: Open and Advanced Large-Scale Video Generative Models

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025. 1, 3, 6, 7, 8

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  52. [52]

    Lavie: High-quality video generation with cascaded latent diffusion mod- els.International Journal of Computer Vision, 133(5): 3059–3078, 2025

    Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion mod- els.International Journal of Computer Vision, 133(5): 3059–3078, 2025. 7, 8

    work page 2025

  53. [53]

    Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks

    Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo. Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2364–2373,

    work page

  54. [54]

    VideoGPT: Video Generation using VQ-VAE and Transformers

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers.arXiv preprint arXiv:2104.10157, 2021. 5, 7

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  55. [55]

    Cogvideox: Text-to-video diffusion models with an expert transformer.ICLR 2025, 2025

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.ICLR 2025, 2025. 1, 3, 6

    work page 2025

  56. [56]

    Magvit: Masked generative video transformer

    Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Haupt- mann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10459–10469,

    work page

  57. [57]

    Generating videos with dynamics-aware implicit generative adversarial networks.arXiv preprint arXiv:2202.10571, 2022

    Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks.arXiv preprint arXiv:2202.10571, 2022. 5, 6, 7

    work page arXiv 2022

  58. [58]

    Video probabilistic diffusion models in pro- jected latent space

    Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in pro- jected latent space. InProceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pages 18456–18466, 2023. 5, 6, 7

    work page 2023

  59. [59]

    Fast video generation with sliding tile attention.arXiv preprint arXiv:2502.04507, 2025

    Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhengzhong Liu, and Hao Zhang. Fast video generation with sliding tile attention.arXiv preprint arXiv:2502.04507, 2025. 1

    work page arXiv 2025

  60. [60]

    Faster video diffusion with trainable sparse attention.arXiv e-prints, pages arXiv–2505,

    Peiyuan Zhang, Haofeng Huang, Yongqi Chen, Will Lin, Zhengzhong Liu, Ion Stoica, Eric P Xing, and Hao Zhang. Faster video diffusion with trainable sparse attention.arXiv e-prints, pages arXiv–2505,

    work page

  61. [61]

    Pointodyssey: A large-scale synthetic dataset for long-term point track- ing

    Yang Zheng, Adam W Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point track- ing. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 19855–19865,

    work page

  62. [62]

    Open-Sora: Democratizing Efficient Video Production for All

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chen- hui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratiz- ing efficient video production for all.arXiv preprint arXiv:2412.20404, 2024. 1, 2, 5 A. Theoretical Proof of Matrix Attention This section derives the attention maps of our Matrix At- tention and compare it with ...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  63. [63]

    and FVMD [28]. FVD computes the Fréchet distance between feature distributions of real and generated videos, where features are extracted using a pretrained I3D net- work [3]; it reflects both overall video quality and tem- poral coherence. FVMD focuses specifically on motion consistency: it tracks keypoints using a pretrained PIPs++ model [61] to obtain ...

    work page