Pith. sign in

REVIEW 3 major objections 6 minor 5 cited by

LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexity

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By replacing self-attention with a linear-complexity MATE block, LinGen makes high-resolution minute-length text-to-video generation practical on a single GPU, with quality comparable to leading commercial models.

desk verdict A serious linear-complexity video backbone whose efficiency claims are solid, but whose load-bearing quality parity with DiT is only shown at an early training checkpoint and with mismatched model widths. read the letter →

arxiv 2412.09856 v2 pith:6KCCP3Y6 submitted 2024-12-13 cs.CV cs.AIcs.LGeess.IV

Hongjie Wang , Chih-Yao Ma , Yen-Cheng Liu , Ji Hou , Tao Xu , Jialiang Wang , Felix Juefei-Xu , Yaqiao Luo
show 5 more authors
Peizhao Zhang Tingbo Hou Peter Vajda Niraj K. Jha Xiaoliang Dai
This is my paper · ORCID
classification cs.CVcs.AIcs.LGeess.IV
keywords text-to-videogenerationlinearcomputationalcomplexityDiffusionTransformersMamba2statespacemodelstemporalattentionminute-lengthvideoMATEblock
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the quadratic cost of self-attention in Diffusion Transformers can be replaced by a linear-complexity block without hurting video quality, and that this makes high-resolution minute-length text-to-video generation possible on a single GPU. It proposes LinGen, which swaps every self-attention layer for a MATE block built from a bidirectional Mamba2 branch and a temporal Swin attention branch. The significance for a general reader is concrete: if the claim holds, long-form video generation stops being a compute privilege of large clusters and becomes a problem whose cost grows linearly with the number of pixels.

What carries the argument

MATE is the load-bearing object: a linear-complexity replacement for self-attention with two branches. The MA-branch stacks a bidirectional Mamba2 block with Rotary-Major Scan (four alternating 3D-to-1D token rearrangements across layers) and review tokens (average-pooled sequence summaries prepended to Mamba2's hidden state). The TE-branch is TEmporal Swin Attention, a shifted-window attention block that gathers short-range spatial and medium-range temporal correlations. The block's job is to keep the generated video consistent across frames while avoiding the adjacency loss and long-range decay that native Mamba introduces.

What would settle it

Rerun the human A/B evaluation at matched model capacity (same hidden dimension and head count for DiT) and after a longer training schedule; if DiT-4B matches or beats LinGen-4B, the win-rate claim becomes a training-efficiency artifact. For the complexity claim, measure FLOPs and latency while doubling both resolution and duration beyond 68 seconds; if growth is superlinear in the number of pixels, the linear-complexity claim fails in the regime it targets.

Watch

Extended reading notes

Core claim

LinGen-4B, a 4-billion-parameter model, generates 512p videos up to 68 seconds at 16 fps on a single H100 GPU. The central experimental claim is that replacing DiT self-attention with MATE raises the human-evaluation win rate to 75.6% over a DiT-4B baseline while cutting FLOPs up to 15x and latency up to 11.5x. Against commercial systems, LinGen-4B earns 50.5%, 52.1%, and 49.1% win rates against Gen-3, LumaLabs, and Kling, which the paper reads as comparable quality. The mechanism is that MATE covers short-, medium-, and long-range correlations at linear cost: Mamba2 handles global structure, Rotary-Major Scan preserves spatial and temporal adjacency, review tokens offset long-range decay, and TESA handles adjacent temporal windows.

Load-bearing premise

The comparison that isolates MATE's effect treats DiT-4B (hidden dimension 3072, 24 heads) as a fair self-attention baseline for LinGen-4B (hidden dimension 2560, 20 heads), with the headline quality win measured early in training at 256p and 40K steps.

Editorial extensions

If this is right

  • Generating 512p, 68-second video on a single H100 becomes routine; the 15x FLOPs and 11.5x latency savings grow with video length and resolution.
  • MATE can be dropped into existing DiT-based video models as a self-attention substitute, transferring the linear-scaling benefit to models already trained in the quadratic regime.
  • The 75.6% win rate over DiT-4B at the same data budget implies the architecture learns text-to-video generation from longer token sequences faster, not just cheaper.
  • LinGen's win rates near 50% against Gen-3, LumaLabs, and Kling suggest that minute-length generation no longer trades away the quality bar set by short commercial clips.
  • Because MATE is orthogonal to sampling distillation, combining it with step-reduction methods could produce a further multiplicative speed-up.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the linear scaling holds beyond 68 seconds, hour-length generation on a single GPU becomes a plausible engineering target, though the paper only demonstrates up to 68 seconds and does not test hour-scale consistency.
  • The RMS and TESA patterns are general techniques for adapting causal sequence models to 2D and 3D grids; they could transfer to other SSM-based image, audio, or multimodal generative models that face the same adjacency problem.
  • A matched-capacity comparison (same hidden dimension and head count for the DiT baseline) would determine how much of the reported quality advantage comes from MATE itself versus from the baseline's training-efficiency profile.
  • Testing MATE as a drop-in for self-attention in an already-trained short-video model, without re-training from scratch, would reveal whether the block can be adopted incrementally.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes LinGen, a text-to-video diffusion framework that replaces the quadratic-complexity self-attention layers of a DiT with a linear-complexity MATE block. MATE consists of an MA-branch (bidirectional Mamba2 with a proposed Rotary-Major Scan and review tokens) and a TE-branch (a temporal Swin attention block, TESA). The authors claim that this architecture achieves up to 15x FLOPs and 11.5x latency reductions over a DiT baseline at 512p/68s while preserving generation quality, and that LinGen-4B is comparable to commercial models such as Gen-3, LumaLabs, and Kling in human evaluation. The paper reports FLOPs formulas (Eqs. 1-3), latency measurements on an H100, VBench scores, ablation studies, and human-evaluation win rates, with the central claimed contribution being 'high-resolution minute-length video generation on a single GPU without compromising quality.'

Significance. If the quality-parity claim were fully supported, this would be a substantive efficiency contribution to video diffusion: the MATE block is a concrete, linear-complexity replacement for self-attention, and the paper's efficiency evidence is comparatively strong. The FLOPs expressions in Eqs. (1)-(3) are explicit and plausible, the H100 latency measurements at 512p/17s/34s/68s are consistent with the claimed complexity scaling, and the ablations (Table 2, Figs. 11-12) provide useful attribution for TESA, RMS, review tokens, and hybrid training. The weakness is in the quality evidence for the central architectural claim: the only direct LinGen-4B vs DiT-4B quality comparison is at an early training checkpoint with a width-mismatched baseline, and the comparison against commercial models is reported without a full human-evaluation protocol. These issues are addressable in revision, so the contribution is defensible but not yet fully established.

major comments (3)
  1. [Sec. 4.3, Fig. 8] The controlled MATE-vs-self-attention comparison (LinGen-4B vs DiT-4B) is performed at 256p after only 40K training steps, and Sec. 4.4 explicitly explains LinGen's advantage by its faster adaptation to longer token sequences, with human evaluations taken at 1K-step checkpoints. This is a training-dynamics measurement rather than a final-quality measurement. The paper never reports a human evaluation of this pair after convergence (e.g., after the 125K-step 256p stage or after the full 512p recipe), so the possibility that DiT catches up or surpasses at convergence is untested. Because the abstract's 'without compromising quality' claim rests on MATE matching self-attention at final quality, please add a late-checkpoint human evaluation with matched training data and compute.
  2. [Sec. 4.1 (Models)] The DiT-4B baseline is not matched in model capacity: DiT-4B uses hidden dimension 3072 with 24 heads, while LinGen-4B uses hidden dimension 2560 with 20 heads. Consequently, the headline efficiency numbers (15x FLOPs, 11.5x latency in Fig. 6) and the Fig. 8 win rates do not isolate the architectural effect of the MATE block; they also include a width and head-count difference. Please report a matched-width DiT baseline, or otherwise control for parameter count and width, in both the efficiency and quality comparisons.
  3. [Sec. 4.3 (Human Evaluation Results)] The human-evaluation protocol is under-specified: the paper reports win rates and states that 'the variance of human evaluation is 3%,' but it does not state the number of evaluators, number of prompts, number of videos per model, tie handling, or confidence intervals. Without this information, near-50% win rates against Gen-3, LumaLabs, and Kling do not by themselves establish 'comparable quality.' Please provide the full protocol and the relevant statistics, or soften the claim accordingly.
minor comments (6)
  1. [Sec. 4.1] The baseline name 'Runaway Gen3' should be 'Runway Gen3'.
  2. [Sec. 4.3] The word 'incidates' should be 'indicates'; also 'commerical' should be 'commercial.'
  3. [Fig. 8 caption / Sec. 4.4] The caption contains 'adapation' and 'faster adapation'; these should be corrected.
  4. [Supp. Table 7] The caption refers to 'the pre-training recipe of LVGen'; this should be 'LinGen.'
  5. [Fig. 6] The axis labels such as '256!,17s' appear garbled in the rendered version and should be formatted as '256p, 17s' (similarly for other resolution/length labels).
  6. [Eq. (3)] The products (T/Tw)(H/Sw)(W/Sw) implicitly assume exact divisibility; please clarify that these are integer ceiling/floor counts of windows.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central linear-complexity and quality claims are empirically measured and externally benchmarked; only minor, non-load-bearing same-team citations appear.

full rationale

LinGen is an empirical systems paper rather than a derivation whose conclusions are encoded in its premises. The headline claims — linear scaling, 15x/11.5x efficiency savings, and quality parity — are supported by measured per-resolution/length FLOPs and latency curves (Fig. 6), human win rates against the authors' own DiT-4B (Fig. 8) and against Gen-3/LumaLabs/Kling (Fig. 9), and VBench-Long scores (Table 1). The FLOPs formulas C_bimamba (Eq. 1), C_RT (Eq. 2), and C_TESA (Eq. 3) are standard per-operation counts that are linear in N by inspection; no parameter is fitted and then renamed a prediction. The DiT-4B baseline is explicitly constructed by the authors: 'We replace MATE blocks in LinGen-4B with global self-attention layers to build a standard DiT' (Sec. 4.1), and the paper itself discloses that the 75.6% win rate is measured 'at 256p after being trained for 40K steps' and speculates it reflects faster adaptation (Secs. 4.3-4.4). That is a real empirical confound — DiT-4B uses hidden dim 3072/24 heads versus LinGen-4B's 2560/20 heads, and convergence-quality parity is untested — but it is a correctness/robustness limitation, not a circular reduction, because the outcome is measured rather than obtained by definition. The same-team citations to Movie Gen [41] for the TAE design, t-schedule, and the loss-curve-quality correlation, and to EMU [5,11] for the quality-tuning recipe, are design borrowings and an ablation heuristic; the central claims do not reduce to those citations. No self-definitional, fitted-input, uniqueness-import, ansatz-smuggling, or renaming step was found. Score 2 reflects the minor non-load-bearing self-citations (notably the loss-curve-quality correlation from [41]) while confirming no definitional circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claims rest on standard complexity formulas, on the validity of the evaluation metrics, and on the training recipe. There are no invented physical or conceptual entities. The free parameters listed are architectural hyperparameters chosen by hand rather than fitted to a target result; they do not function as hidden degrees of freedom in a derivation.

free parameters (3)
  • TESA window sizes (Tw, Sw) = Sw=4; Tw not explicitly listed
    Hand-chosen: a small spatial window keeps TESA linear and cheap; the paper states 'we set it to 4x4 in practice'.
  • Review token pooling ranges (pt, py, px) = {8, 4, 4}
    Hand-set as stated in Eq. 2 to keep the extra FLOPs marginal; not optimized or fitted to data.
  • Mamba2 hidden state size ds = 128 for LinGen w/ Mamba2; 16 for LinGen w/ Mamba
    Chosen following Mamba2 default recommendations and to balance representational capacity against latency, not from a fitting procedure.
assumptions (3)
  • standard math The FLOPs formulas for bidirectional Mamba2 (Eq. 1 and Supp. Eq. 4-6) and for windowed attention (Eq. 3) correctly capture computational complexity.
    The linear-in-N scaling of state-space models and windowed attention is established in the cited works [6,31]; the paper derives the terms from these definitions.
  • domain assumption VBench and human A/B win rates are valid proxies for video generation quality.
    The paper leans on VBench-Long, VBench-Custom, and human preference to support the 'comparable quality' claim. The supplementary text itself notes VBench does not perfectly align with human preference, so reliance on these metrics is a genuine assumption.
  • domain assumption The progressive training recipe with 24M licensed videos and quality tuning is sufficient to reach the reported quality.
    The final model's quality depends on this specific recipe, dataset, and compute budget. The paper provides no external replication or controlled comparison on a public dataset.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexity." pith.science (2026). https://pith.science/paper/6KCCP3Y6

@misc{pith2026241209856,
  author       = {Pith},
  title        = {Pith review of: LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6KCCP3Y6}},
  note         = {Machine review of arXiv:2412.09856}
}
abstract

Text-to-video generation enhances content creation but is highly computationally intensive: The computational cost of Diffusion Transformers (DiTs) scales quadratically in the number of pixels. This makes minute-length video generation extremely expensive, limiting most existing models to generating videos of only 10-20 seconds length. We propose a Linear-complexity text-to-video Generation (LinGen) framework whose cost scales linearly in the number of pixels. For the first time, LinGen enables high-resolution minute-length video generation on a single GPU without compromising quality. It replaces the computationally-dominant and quadratic-complexity block, self-attention, with a linear-complexity block called MATE, which consists of an MA-branch and a TE-branch. The MA-branch targets short-to-long-range correlations, combining a bidirectional Mamba2 block with our token rearrangement method, Rotary Major Scan, and our review tokens developed for long video generation. The TE-branch is a novel TEmporal Swin Attention block that focuses on temporal correlations between adjacent tokens and medium-range tokens. The MATE block addresses the adjacency preservation issue of Mamba and improves the consistency of generated videos significantly. Experimental results show that LinGen outperforms DiT (with a 75.6% win rate) in video quality with up to 15$\times$ (11.5$\times$) FLOPs (latency) reduction. Furthermore, both automatic metrics and human evaluation demonstrate our LinGen-4B yields comparable video quality to state-of-the-art models (with a 50.5%, 52.1%, 49.1% win rate with respect to Gen-3, LumaLabs, and Kling, respectively). This paves the way to hour-length movie generation and real-time interactive video generation. We provide 68s video generation results and more examples in our project website: https://lineargen.github.io/.

Figures

Figures reproduced from arXiv: 2412.09856 by the authors.

Figure 1
Figure 1. LinGen generates photorealistic high-resolution long videos with linear computational complexity. (a) High-quality videos generated using our LinGen model. (b) The computational cost scaling curves across different video resolutions and lengths. LinGen achieves 15× speed-up compared to the standard DiT when generating 68s-length videos at 512p resolution. work that scales linearly in the number of pixels in gen￾erat… view at source ↗
Figure 2
Figure 2. Overview of the LinGen denoising module. LinGen replaces self-attention layers with a MATE block, which inherits linear complexity from its two branches: MA-branch and TE-branch. The MA-branch consists of a bidirectional Mamba2 block, RMS, and review tokens to cover short-to-long-range correlations. The TE-branch is a TEmporal Swin Attention block that addresses the adjacency preservation issue and improves the cons… view at source ↗
Figure 3
Figure 3. The bidirectional Mamba2 module. Native Mamba2 only generates the lower triangular part of the attention map due to its causal characteristic. Thus, we deploy bidirectional Mamba2 to obtain the complete attention map for vision tasks. by LLaMa-3.1 [9] as input. Most importantly, LinGen re￾places the self-attention layer of vanilla DiTs with our pro￾posed MATE block, achieving linear computational com￾plexity. MATE i… view at source ↗
Figures from the paper (15 more)
Figure 5
Figure 5. Figure 5: TEmporal Swin Attention (TESA). We divide the to￾ken tensor into small windows and calculate self-attention within each window. The windows are alternately shifted across layers to cross the boundaries of local windows. The window size remains fixed across different re…
Figure 6
Figure 6. Figure 6: Computational cost comparison between DiT-4B and LinGen-4B. (a) Latency. (b) FLOPs. The cost of LinGen scales significantly slower with both video length and video resolution than DiT. Latency is measured on a single H100 GPU. model, OpenSora [76]. We provide compariso…
Figure 7
Figure 7. Figure 7: Visual examples of videos generated from different models. LinGen-4B generates videos that have similar quality to state-of [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 12
Figure 12. Figure 12: Win rates of human evaluation on quality comparison [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]
Figure 9
Figure 9. Figure 9: Win rates of human evaluation on the quality and text [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: LinGen adapts much faster to the new task than DiT. [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Loss curves of 256p text-to-video pre-training under [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 13
Figure 13. Figure 13: Average minimum distance between adjacent tokens. [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 14
Figure 14. Figure 14: Examples of 17-second and 68-second videos generated by LinGen. [PITH_FULL_IMAGE:figures/full_fig_p013_14.png]
Figure 15
Figure 15. Figure 15: Comparisons with typical open-source video generative models. [PITH_FULL_IMAGE:figures/full_fig_p014_15.png]
Figure 16
Figure 16. Figure 16: Comparisons with state-of-the-art accessible commercial models. [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]
Figure 17
Figure 17. Figure 17: Comparisons with existing trials on generating minute-length videos. [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]
Figure 18
Figure 18. Figure 18: Visual examples of ablation experiments on the TESA block, RMS, and review tokens. [PITH_FULL_IMAGE:figures/full_fig_p017_18.png]
Figure 19
Figure 19. Figure 19: Visual examples of ablation experiments on hybrid training and quality-tuning. [PITH_FULL_IMAGE:figures/full_fig_p018_19.png]
Figure 21
Figure 21. Figure 21: Latency of generating 512p 17s videos with different [PITH_FULL_IMAGE:figures/full_fig_p018_21.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VMoBA: Mixture-of-Block Attention for Video Diffusion Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    VMoBA is a sparse attention mechanism for video diffusion models that combines cyclic 1D-2D-3D block partitioning with global and threshold-based block selection to reduce training FLOPs while keeping generation quality.

  2. M4V: Multimodal Mamba for Efficient Text-to-Video Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    M4V shows a Mamba-based text-to-video model can roughly match attention-based PyramidFlow on VBench while cutting mixer-layer FLOPs by 45% at 768x1280.

  3. Exploring Diffusion Transformer Designs via Grafting

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Grafting uses activation distillation and lightweight fine-tuning to edit pretrained diffusion transformers into hybrid architectures with near-baseline quality at under 2% pretraining compute.

  4. Video World Models with Long-term Spatial Memory

    cs.CV 2025-06 conditional novelty 6.0 of 10

    An autoregressive video world model with a persistent static point-cloud spatial memory and sparse episodic keyframes improves revisit consistency over point-cloud-conditioned baselines.

  5. Long-Context State-Space Video World Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A hybrid state-space and local-attention architecture gives autoregressive video diffusion models long-term spatial memory with constant per-frame inference cost, demonstrated on Maze and Minecraft.

Reference graph

Works this paper leans on

76 extracted references · 26 canonical work pages · cited by 5 Pith papers

  1. [1]

    Video generation models as world simu- lators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simu- lators. https://openai.com/research/video- generation - models - as - world - simulators,

    work page

  2. [2]

    VideoCrafter1: Open diffusion models for high-quality video generation

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. VideoCrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512, 2023. 1, 3, 8

    arXiv

  3. [3]

    VideoCrafter2: Overcoming data limitations for high-quality video diffu- sion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. VideoCrafter2: Overcoming data limitations for high-quality video diffu- sion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7310– 7320, 2024. 2, 6, 8

    work page 2024

  4. [4]

    DiffEdit: Diffusion-based seman- tic image editing with mask guidance

    Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. DiffEdit: Diffusion-based seman- tic image editing with mask guidance. arXiv preprint arXiv:2210.11427, 2022. 1

    arXiv 2022

  5. [5]

    EMU: Enhanc- ing image generation models using photogenic needles in a haystack

    Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xi- aofang Wang, Abhimanyu Dubey, et al. EMU: Enhanc- ing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807, 2023. 1, 5, 6

    arXiv 2023

  6. [6]

    Transformers are SSMs: General- ized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are SSMs: General- ized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024. 2, 4, 8, 3, 5

    arXiv 2024

  7. [7]

    FlashAttention: Fast and memory-efficient exact attention with IO-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christo- pher R ´e. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. Advances in Neural Informa- tion Processing Systems, 35:16344–16359, 2022. 3

    work page 2022

  8. [8]

    Alabdul- mohsin, et al

    Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M. Alabdul- mohsin, et al. Patch n’Pack: NaViT, a vision transformer for any aspect ratio and resolution. Advances in Neural In- formation Processing Systems, 36, 2024. 2

    work page 2024

Show all 76 references
  1. [9]

    The LLaMa 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The LLaMa 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  2. [10]

    Matten: Video generation with Mamba- attention

    Yu Gao, Jiancheng Huang, Xiaopeng Sun, Zequn Jie, Yujie Zhong, and Lin Ma. Matten: Video generation with Mamba- attention. arXiv preprint arXiv:2405.03025, 2024. 1, 3, 4

    2024 arXiv

  3. [11]

    EMU Video: Factoriz- ing text-to-video generation by explicit image conditioning

    Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Du- val, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. EMU Video: Factoriz- ing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709, 2023. 1, 5

    2023 arXiv

  4. [12]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. 1, 2

    2023 arXiv

  5. [13]

    HiPPO: Recurrent memory with optimal polyno- mial projections

    Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christo- pher R ´e. HiPPO: Recurrent memory with optimal polyno- mial projections. Advances in Neural Information Process- ing Systems, 33:1474–1487, 2020. 2

    2020

  6. [14]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher R ´e. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021. 2

    2021 arXiv

  7. [15]

    MambaAD: Exploring state space models for multi-class unsupervised anomaly detec- tion

    Haoyang He, Yuhu Bai, Jiangning Zhang, Qingdong He, Hongxu Chen, Zhenye Gan, Chengjie Wang, Xiangtai Li, Guanzhong Tian, and Lei Xie. MambaAD: Exploring state space models for multi-class unsupervised anomaly detec- tion. arXiv preprint arXiv:2404.06564, 2024. 1

    2024 arXiv

  8. [16]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020. 1

    2020

  9. [17]

    Kingma, Ben Poole, Mohammad Norouzi, David J

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, et al. Imagen Video: High definition video generation with diffusion mod- els. arXiv preprint arXiv:2210.02303, 2022. 3

    arXiv

  10. [18]

    CogVideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. CogVideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022. 8

    2022 arXiv

  11. [19]

    ZigMa: Zigzag Mamba diffusion model

    Vincent Tao Hu, Stefan Andreas Baumann, Ming Gui, Olga Grebenkova, Pingchuan Ma, Johannes Fischer, and Bjorn Ommer. ZigMa: Zigzag Mamba diffusion model. arXiv preprint arXiv:2403.13802, 2024. 1, 4

    2024 arXiv

  12. [20]

    VBench: Com- prehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Com- prehensive benchmark suite for video generative models....

    2024

  13. [21]

    MiraData: A large-scale video dataset with long durations and structured captions

    Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xin- tao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. MiraData: A large-scale video dataset with long durations and structured captions. arXiv preprint arXiv:2407.06358 ,

    arXiv

  14. [22]

    Imagic: Text-based real image editing with diffusion models

    Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007–6017, 2023. 1

    2023

  15. [23]

    BK-SDM: A lightweight, fast, and cheap version of stable diffusion

    Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells, and Shinkook Choi. BK-SDM: A lightweight, fast, and cheap version of stable diffusion. arXiv preprint arXiv:2305.15798, 2023. 3

    2023 arXiv

  16. [24]

    Kling AI: Next-generation AI creative studio

    Kling AI. Kling AI: Next-generation AI creative studio. https://klingai.com/, 2024. 2, 5, 7, 1, 4 9

    2024

  17. [25]

    VideoPoet: A large language model for zero-shot video gen- eration

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos ´e Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vigh- nesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. VideoPoet: A large language model for zero-shot video gen- eration. arXiv preprint arXiv:2312.14125, 2023. 2

    2023 arXiv

  18. [26]

    Pika labs

    Pika Labs. Pika labs. https://www.pika.art/, 2024. 8

    2024

  19. [27]

    xFormers: A modular and hackable trans- former modelling library

    Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xFormers: A modular and hackable trans- forme...

    2022

  20. [28]

    T2V- Turbo: Breaking the quality bottleneck of video consis- tency model with mixed reward feedback

    Jiachen Li, Weixi Feng, Tsu-Jui Fu, Xinyi Wang, Sug- ato Basu, Wenhu Chen, and William Yang Wang. T2V- Turbo: Breaking the quality bottleneck of video consis- tency model with mixed reward feedback. arXiv preprint arXiv:2405.18750, 2024. 1, 3, 6

    2024 arXiv

  21. [29]

    T2V- Turbo-v2: Enhancing video generation model post-training through data, reward, and conditional guidance design.arXiv preprint arXiv:2410.05677, 2024

    Jiachen Li, Qian Long, Jian Zheng, Xiaofeng Gao, Robinson Piramuthu, Wenhu Chen, and William Yang Wang. T2V- Turbo-v2: Enhancing video generation model post-training through data, reward, and conditional guidance design.arXiv preprint arXiv:2410.05677, 2024. 8

  22. [30]

    Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le

    Yaron Lipman, Ricky T.Q. Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling. arXiv preprint arXiv:2210.02747, 2022. 3

    2022 arXiv

  23. [31]

    Swin Transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021. 5

    2021

  24. [32]

    VDT: General-purpose video diffusion transformers via mask modeling

    Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, and Mingyu Ding. VDT: General-purpose video diffusion transformers via mask modeling. arXiv preprint arXiv:2305.13311, 2023. 3

    2023 arXiv

  25. [33]

    Dream machine

    Luma Labs. Dream machine. https://lumalabs.ai/ dream-machine, 2024. 2, 5, 1

    2024

  26. [34]

    Diffusion probabilistic models for 3D point cloud generation

    Shitong Luo and Wei Hu. Diffusion probabilistic models for 3D point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2837–2845, 2021. 1

    2021

  27. [35]

    Latent Consistency Models: Synthesizing high- resolution images with few-step inference

    Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent Consistency Models: Synthesizing high- resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023. 3

    2023 arXiv

  28. [36]

    On distillation of guided diffusion models

    Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 14297–14306, 2023. 3

    2023

  29. [37]

    Scaling diffusion Mamba with bidirectional SSMs for efficient image and video gener- ation

    Shentong Mo and Yapeng Tian. Scaling diffusion Mamba with bidirectional SSMs for efficient image and video gener- ation. arXiv preprint arXiv:2405.15881, 2024. 1, 3

    2024 arXiv

  30. [38]

    Transframer: Arbitrary frame prediction with generative models

    Charlie Nash, Joao Carreira, Jacob Walker, Iain Barr, An- drew Jaegle, Mateusz Malinowski, and Peter Battaglia. Transframer: Arbitrary frame prediction with generative models. arXiv preprint arXiv:2203.09494, 2022. 2

    2022 arXiv

  31. [39]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  32. [40]

    SDXL: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 1

    2023 arXiv

  33. [41]

    Movie Gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie Gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  34. [42]

    RawFilm: 8k cinematic royalty-free stock footage

    RawFilm, Inc. RawFilm: 8k cinematic royalty-free stock footage. https://raw.film/, 2024. 6

    2024

  35. [43]

    SAMBA: Simple hybrid state space models for efficient unlimited context language mod- eling

    Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. SAMBA: Simple hybrid state space models for efficient unlimited context language mod- eling. arXiv preprint arXiv:2406.07522, 2024. 6

    2024 arXiv

  36. [44]

    MambaCSR: Dual-interleaved scanning for compressed image super-resolution with SSMs

    Yulin Ren, Xin Li, Mengxi Guo, Bingchen Li, Shijie Zhao, and Zhibo Chen. MambaCSR: Dual-interleaved scanning for compressed image super-resolution with SSMs. arXiv preprint arXiv:2408.11758, 2024. 1

    2024 arXiv

  37. [45]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022. 1

    2022

  38. [46]

    Introducing Gen-3 alpha

    Runway ML. Introducing Gen-3 alpha. https : / / runwayml . com / research / introducing - gen - 3-alpha, 2024. 2, 5, 7, 1, 4, 8

    2024

  39. [47]

    Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informatio...

    2022

  40. [48]

    GLU variants improve transformer

    Noam Shazeer. GLU variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. 2

    2002 arXiv

  41. [49]

    Emu Edit: Precise image editing via recognition and gen- eration tasks

    Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu Edit: Precise image editing via recognition and gen- eration tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pa...

    2024

  42. [50]

    NormFormer: Improved transformer pretraining with extra normalization

    Sam Shleifer, Jason Weston, and Myle Ott. NormFormer: Improved transformer pretraining with extra normalization. arXiv preprint arXiv:2110.09456, 2021. 2

    2021 arXiv

  43. [51]

    Shutterstock: Stock photos, royalty-free images, graphics, vectors, videos, and music

    Shutterstock, Inc. Shutterstock: Stock photos, royalty-free images, graphics, vectors, videos, and music. https:// www.shutterstock.com/. 6

  44. [52]

    Make-a-Video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-Video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,

    arXiv

  45. [53]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Confer- ence on Machine Learning, pages 2256–2265. PMLR, 2015. 1

    2015

  46. [54]

    Consistency models

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023. 3

    2023 arXiv

  47. [55]

    UL2: Unifying language learning paradigms

    Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Ja- son Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shak- eri, Dara Bahri, Tal Schuster, et al. UL2: Unifying language learning paradigms. arXiv preprint arXiv:2205.05131, 2022. 3

    2022 arXiv

  48. [56]

    Genmo Team. Mochi 1. https: // github. com/ genmoai/models, 2024. 4

    2024

  49. [57]

    DiM: Diffusion Mamba for efficient high-resolution image synthesis

    Yao Teng, Yue Wu, Han Shi, Xuefei Ning, Guohao Dai, Yu Wang, Zhenguo Li, and Xihui Liu. DiM: Diffusion Mamba for efficient high-resolution image synthesis. arXiv preprint arXiv:2405.14224, 2024. 2

    2024 arXiv

  50. [58]

    LION: Latent point dif- fusion models for 3D shape generation

    Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, Karsten Kreis, et al. LION: Latent point dif- fusion models for 3D shape generation. Advances in Neural Information Processing Systems, 35:10021–10039, 2022. 1

    2022

  51. [59]

    Phenaki: Variable length video generation from open domain textual descriptions

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kin- dermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Le...

    2022

  52. [60]

    An empirical study of Mamba-based language models

    Roger Waleffe, Wonmin Byeon, Duncan Riach, Bran- don Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of Mamba-based language models. arXiv preprint arXiv:2406.07887, 2024. 3

    2024 arXiv

  53. [61]

    Jha, and Yuchen Liu

    Hongjie Wang, Difan Liu, Yan Kang, Yijun Li, Zhe Lin, Ni- raj K. Jha, and Yuchen Liu. Attention-driven training-free efficiency enhancement of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 16080–16089, 2024. 3

    2024

  54. [62]

    ModelScope text-to-video technical report

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. ModelScope text-to-video technical report. arXiv preprint arXiv:2308.06571, 2023. 1, 2, 3, 6, 8

    2023 arXiv

  55. [63]

    VideoLCM: Video latent consistency model

    Xiang Wang, Shiwei Zhang, Han Zhang, Yu Liu, Yingya Zhang, Changxin Gao, and Nong Sang. VideoLCM: Video latent consistency model. arXiv preprint arXiv:2312.09109,

    arXiv

  56. [64]

    LA VIE: High-quality video genera- tion with cascaded latent diffusion models

    Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. LA VIE: High-quality video genera- tion with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023. 3, 6, 8

    2023 arXiv

  57. [65]

    Loong: Generating minute-level long videos with autoregressive lan- guage models

    Yuqing Wang, Tianwei Xiong, Daquan Zhou, Zhijie Lin, Yang Zhao, Bingyi Kang, Jiashi Feng, and Xihui Liu. Loong: Generating minute-level long videos with autoregressive lan- guage models. arXiv preprint arXiv:2410.02757, 2024. 3, 1

    2024 arXiv

  58. [66]

    Progressive autoregressive video diffusion models

    Desai Xie, Zhan Xu, Yicong Hong, Hao Tan, Difan Liu, Feng Liu, Arie Kaufman, and Yang Zhou. Progressive autoregressive video diffusion models. arXiv preprint arXiv:2410.08151, 2024. 3, 1

    2024 arXiv

  59. [67]

    Demystify- ing clip data

    Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystify- ing clip data. arXiv preprint arXiv:2309.16671, 2023. 3

    2023 arXiv

  60. [68]

    ByT5: Towards a token-free future with pre-trained byte- to-byte models

    Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raf- fel. ByT5: Towards a token-free future with pre-trained byte- to-byte models. Transactions of the Association for Compu- tational Linguistics, 10:291–306, 2022. 3

    2022

  61. [69]

    Dif- fusion models without attention

    Jing Nathan Yan, Jiatao Gu, and Alexander M Rush. Dif- fusion models without attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8239–8249, 2024. 2

    2024

  62. [70]

    VideoGPT: Video generation using VQ-V AE and transformers

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. VideoGPT: Video generation using VQ-V AE and transformers. arXiv preprint arXiv:2104.10157, 2021. 2

    2021 arXiv

  63. [71]

    Paint by example: Exemplar-based image editing with diffusion mod- els

    Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 18381–18391,

  64. [72]

    CogVideoX: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. CogVideoX: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 6, 1, 4

    2024 arXiv

  65. [73]

    Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, et al

    Lijun Yu, Yong Cheng, Kihyuk Sohn, Jos ´e Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, et al. MagViT: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, ...

    2023

  66. [74]

    Root mean square layer nor- malization

    Biao Zhang and Rico Sennrich. Root mean square layer nor- malization. Advances in Neural Information Processing Sys- tems, 32, 2019. 2

    2019

  67. [75]

    Show-1: Marrying pixel and latent diffusion models for text-to-video generation

    David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. International Journal of Com- puter Vision, pages 1–15, 2024. 8

    2024

  68. [76]

    Open-Sora: Democratizing efficient video production for all

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-Sora: Democratizing efficient video production for all. https://github.com/hpcaitech/Open- Sora, 2024. 1, 2, 6, 7, 4, 8 11 LinGen: Towards High-Resolutio...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.