Pith. sign in

REVIEW 4 major objections 5 minor 65 references

Factorized Video Autoencoders for Efficient Generative Modelling

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Four-plane video latents cut diffusion sequence length from t·h·w to t·(h+w)+2·h·w, and the paper shows this compressed latent still yields generation FVD on par with the volumetric W.A.L.T. baseline while training roughly twice as fast.

desk verdict Four-plane factorization delivers a real ~2x speedup for latent video diffusion, but the unexplained 7-point FVD gap in the reimplemented W.A.L.T. baseline keeps the quality-parity claim from being fully established. read the letter →

arxiv 2412.04452 v2 pith:BJXY2R7B submitted 2024-12-05 cs.CV

Mohammed Suhail , Carlos Esteves , Leonid Sigal , Ameesh Makadia This is my paper · ORCID
classification cs.CV
keywords videogenerationlatentdiffusionfactorizedrepresentationautoencodertri-planetransformertokenizationefficientgenerativemodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a video autoencoder can compress the volumetric latent t×h×w into four axis-aligned planes—two spatial and two spatiotemporal—whose combined size t×(h+w)+2×h×w grows sublinearly with the input volume, and that this compact latent still carries enough information for a transformer-based latent diffusion model to generate videos of quality comparable to the volumetric baseline. The claim matters because video diffusion is bottlenecked by sequence length: if the factorization preserves fidelity, it offers a drop-in efficiency upgrade for class-conditional generation, frame prediction, and video interpolation. The paper reports reconstruction PSNR of 27.11 vs 27.64 for the volumetric W.A.L.T. baseline at 128×128 with 17 frames, and generation FVD of 38 vs 39 on UCF-101 for the re-implemented W.A.L.T.* baseline, while cutting training iteration time from 750 ms to 380 ms.

What carries the argument

The machinery is the four-plane factorization of the latent volume Z∈$R^{{t×h×w×c}}$: two spatial planes $P^{1}$_xy, $P^{2}$_xy∈$R^{{h×w×c}}$ come from splitting the time axis and mean-pooling each half, while two spatiotemporal planes P_xt∈$R^{{t×h×c}}$ and P_yt∈$R^{{t×w×c}}$ come from mean-pooling along width and height. The decoder recomposes the volume by querying the four planes at each (x,y,t) and concatenating the features, then feeds the rebuilt volume through a 3D CNN decoder. This factorization is what turns the transformer sequence of length t×h×w into one of length t×(h+w)+2×h×w, and the paper's ablations show concatenation over summation and mean pooling over learned linear projection as the best choices.

What would settle it

Train the same transformer diffusion model on the volumetric latent with the same data and compute, and measure FVD on UCF-101 128×128: if the volumetric model matches or beats the factorized model's 38—or if the original W.A.L.T.'s published 46 is reproduced instead of the 39 reported for W.A.L.T.*—then the claim that four-plane factorization preserves generation quality is undermined. The paper's own Table 4 offers a second check: swapping mean pooling for linear projection worsens FVD to 50, showing the result depends on that design choice.

Watch

Extended reading notes

Core claim

The paper's central claim is that volumetric video latents can be factorized into four planes—two temporal-mean spatial planes and two spatial-mean temporal planes—and rebuilt by concatenating the four queried plane features at each voxel, losing only a small amount of reconstruction fidelity while almost halving the diffusion sequence length (1280 to 672 tokens at 128×128). The paper shows this factorized space supports a transformer-based latent diffusion model whose generation FVD is on par with the volumetric baseline on UCF-101 class-conditional generation (38 vs 39 for W.A.L.T.* at 128×128, and 58.27 vs 84.68 at 256×256 in the factorized model's favor), while on K600 frame prediction it trails (8.6 vs 5.7). The four-plane design is framed as a fix for tri-plane latents' information mixing: two spatial planes retain more spatial structure and make frame-conditional tasks straightforward.

Load-bearing premise

The re-trained W.A.L.T. baseline (W.A.L.T.*) faithfully represents the original W.A.L.T. model, even though its FVD differs from the published number (39 vs 46 on UCF-101 128×128).

Editorial extensions

If this is right

  • At 128×128, the diffusion sequence length drops from 1280 to 672, giving roughly 2× faster training iterations (380 ms vs 750 ms) and 2.4× faster inference per video (0.17 s vs 0.40 s).
  • At 256×256, the sequence length stays at 672 while the volumetric baseline grows to 1280, and generation FVD improves over W.A.L.T.* (58.27 vs 84.68 on UCF-101).
  • The same factorized autoencoder supports class-conditional generation, frame prediction (using one spatial plane as conditioning), and video interpolation (conditioning on both boundary-frame spatial planes), with interpolation FVD 156.1 on DAVIS-7 vs 199.3 for VIDIM.
  • Longer videos become feasible: 56-frame generation runs at 378 ms per step while the volumetric baseline exceeds memory limits.
  • The factorization holds in both AE and VAE settings (with W.A.L.T.-style and WF-VAE-style autoencoders), so the efficiency gain is not tied to a particular autoencoder loss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The sublinear scaling suggests the factorization becomes more attractive for longer and higher-resolution videos; the 56-frame experiment hints at this, but the paper does not test 128-frame or higher-resolution regimes.
  • Appendix F sketches a joint image-video training strategy that discards one redundant spatial plane for images; if that works, the same tokenizer could unify image and video diffusion training with only a small sequence-length increase (288 vs 256 tokens for a 16×16 latent).
  • Because the spatial planes are time-averaged, the representation may be lossy for fast, high-frequency motion; the comparable FVD on UCF-101, whose actions are mostly slow, does not guarantee the same on datasets with rapid motion, which would be a natural stress test.
  • The factorization and recombination are simple pooling and concatenation operations, so the representation could be ported to other 3D volumetric data domains (e.g., medical volumes or 3D scenes) wherever a transformer operates on a flattened volume.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a four-plane factorized video latent representation, replacing the traditional volumetric latent volume with two spatial planes and two spatiotemporal planes. For a 128x128, 17-frame video, the resulting transformer sequence length drops from 1,280 to 672 (Table 1). The authors evaluate reconstruction quality on Kinetics-600, class-conditional generation on UCF-101, frame prediction on K600, and two-frame interpolation on DAVIS-7/UCF-7, reporting generation FVD values comparable to a reimplemented W.A.L.T. baseline (W.A.L.T.*) while achieving roughly 2x faster diffusion training (380 ms vs 750 ms per iteration). The paper includes ablations of the factorization and combine operations, a tri-plane comparison, and timing measurements across several hardware configurations.

Significance. If the empirical claims hold, this is a useful efficiency contribution: the four-plane factorization preserves reconstruction fidelity and competitive generation quality while substantially shortening the sequence length presented to a transformer-based diffusion model. The paper is a purely empirical study with careful attention to ablations (Sec. 4.5, Appendix D) and hardware timing (Appendix C), which are strengths. However, the central 'without sacrificing quality' claim is not yet securely established. The head-to-head generation comparison rests on a reimplemented W.A.L.T.* baseline whose FVD differs from the published W.A.L.T. by 7 points (39 vs 46, Table 2), with no explanation. In addition, the key UCF-128 comparison is a one-point FVD gap with no error bars, and the frame-prediction result shows a clear degradation against the published W.A.L.T. (8.6 vs 3.3). These issues make the central claim quantitatively fragile despite the clear efficiency benefit.

major comments (4)
  1. [Sec. 4.1 / Table 2] The paper states in Sec. 4.1 that 'we were able to reproduce the model in terms of similar datasets and performance,' but Table 2 reports W.A.L.T.* with FVD 39 on UCF-101 128x128, whereas the original W.A.L.T. paper reports 46. This 7-point gap is unexplained. The central 'comparable to W.A.L.T.' claim is based on the 38 vs 39 comparison against this reimplementation, not against the published system. The authors need to explain the discrepancy (architecture changes, training data, evaluation protocol) or compare against the official W.A.L.T. checkpoint; otherwise the headline quality comparison is not established.
  2. [Sec. 4.2.2 / Table 2] The UCF-128 class-conditional FVD comparison is 38 (4Plane) vs 39 (W.A.L.T.*), a one-point difference. FVD is a distributional metric with sample noise, and no error bars, confidence intervals, or multiple-seed results are reported. The text concludes the results are 'comparable,' but a single one-point difference without variance characterization is not sufficient support. Please report variance across seeds or another statistical quantification.
  3. [Sec. 4.3 / Table 2] For frame prediction on Kinetics-600, the reported FVD is 8.6 for 4Plane, 5.7 for W.A.L.T.*, and 3.3 for the original W.A.L.T. This is a material degradation, yet Sec. 4.3 states the model is 'comparable to W ALT*.' The gap is roughly 50% relative to the original W.A.L.T. and should be discussed honestly, including whether the efficiency gain justifies this quality loss.
  4. [Sec. 4.2.2 / Table 2] The 256x256 result (4Plane FVD 58.27 vs W.A.L.T.* 84.68) is presented as evidence that a shorter sequence 'can in fact reduce the modeling burden on the denoiser network.' This is a strong interpretative claim, but no analysis is provided to support it (e.g., training curves, controlled sequence-length ablations, or diagnostics). Given that the reimplementation itself is suspect at 128x128, this result does not yet provide a robust basis for that conclusion. Please provide additional evidence or temper the claim.
minor comments (5)
  1. [Sec. 4.1] The text contains a typo: 'apendix' should be 'appendix.'
  2. [Table 2] The table note contains typos: 'exeperiments' and 'resolutios' should be 'experiments' and 'resolutions.' The notation 'W ALT*' is also awkward; consider 'W.A.L.T.*' for clarity.
  3. [Sec. 4 (intro)] The section intro states the factorized autoencoder design is used 'without modification across experiments,' but Sec. 3.1.1 defines different constructions for the spatial planes: Eq. (3) uses time-split aggregation for class-conditional generation and frame prediction, while Eq. (4) uses encoded boundary frames for interpolation. Please clarify whether the design changes or only the input source to the planes changes.
  4. [Appendix F] The joint image-video training strategy in Appendix F is explicitly untested ('While we have not experimented with it'). This is fine as speculation, but it should be labeled as a proposed extension rather than a claim about the method's demonstrated capabilities.
  5. [Sec. 4.2.2 / Appendix C] The timing statement '380 ms compared to 750 milliseconds for W.A.L.T.' is ambiguous about batch size. Appendix C says the reported timings correspond to a batch size of 256 but then says they approximately align with the batch-size-8 row of Figure 5. Please make the reported hardware and batch-size conditions internally consistent.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the factorization is a fixed architectural transformation, and all generative claims are evaluated against external datasets and metrics; the unexplained W.A.L.T.* baseline discrepancy is a reproducibility concern, not circularity.

full rationale

The paper's central derivation is an architectural factorization: the encoder produces a volumetric latent Z, which is deterministically projected onto four planes via mean pooling (Eqs. 1-3) and recombined by concatenation into a volume V. This is a fixed, explicitly defined transformation, not a quantity fitted to the evaluation metric. The diffusion model is then trained on the flattened four-plane sequence, and generation quality is measured externally with FVD, Inception Score, PSNR, SSIM, and LPIPS on UCF-101, Kinetics-600, DAVIS-7, and UCF-7. No equation defines a predicted quantity in terms of the fitted data, and no load-bearing argument reduces to a self-citation. The strongest concern raised by the paper's own text is the W.A.L.T.* reimplementation: Section 4.1 claims 'we were able to reproduce the model in terms of similar datasets and performance,' yet Table 2 reports W.A.L.T.* FVD 39 versus the original W.A.L.T. FVD 46, and the Table 2 note defines W.A.L.T.* as 'our re-training and re-evaluation of the W ALT baseline.' This unexplained 7-point discrepancy affects the fairness of the head-to-head comparison, but it is a baseline-fidelity and experimental-comparability issue, not a circular-reasoning issue: the proposed model's outputs are not defined in terms of the baseline's outputs, and the baseline discrepancy does not make the four-plane claim equivalent to its inputs. Design choices such as mean pooling versus linear projection and concatenation versus summation are selected by comparing FVD on the benchmark, which is model selection on held-out evaluation, not circularity. Overall, the derivation chain is self-contained and empirically grounded, with no step where a prediction is forced by construction. The score reflects only the mild, non-circular concern about the unvalidated reimplemented baseline and the absence of released code or checkpoints to verify that baseline's fidelity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard deep learning assumptions: the W.A.L.T. and MAGVIT-v2 architectures as building blocks, FVD as a quality metric, cross-dataset transfer from K600 to UCF-101, and the representativeness of an undisclosed internal dataset. No new physical entities are introduced. The design choices of plane count, pooling type, and combine type are free parameters selected by ablation on the evaluation benchmark.

free parameters (4)
  • Number of planes (4 vs 3) = 4, selected via ablation
    Appendix D tri-plane ablation: four-plane FVD 38 vs tri-plane 52 on UCF-101; the extra spatial plane is a design choice tuned on the target metric.
  • Factorization operation = Mean pooling, selected over learned linear projection
    Table 4: mean pooling gives UCF FVD 38 vs 50 for linear projection; chosen to maximize generation FVD on the same benchmark where final results are reported.
  • Combine operation = Concatenation, selected over summation
    Table 5: concat gives UCF FVD 38 vs 45 and K600 FVD 8.6 vs 27; chosen via ablation.
  • Temporal split ratio for spatial planes = floor(t/2) and ceil(t/2) halves
    Eq 3: the split of the latent volume into two halves for P1_xy and P2_xy is chosen by hand; no ablation is provided.
assumptions (4)
  • domain assumption The pre-trained autoencoder and diffusion training recipes from W.A.L.T. and MAGVIT-v2 are valid building blocks.
    Used throughout; the paper builds directly on these architectures (Sections 2.2, 3.1.1, E.1) without re-deriving their validity.
  • domain assumption FVD on UCF-101 / Kinetics-600 is a reliable proxy for video generation quality at the reported effect sizes.
    FVD is used as the primary metric (Section 4.2); no confidence intervals or repeat runs are provided, so the metric's stability at 1-point differences is assumed.
  • domain assumption The K600-trained autoencoder's factorized latent distribution transfers to UCF-101 for the diffusion model.
    Class-conditional diffusion is trained on UCF-101 with an autoencoder trained on K600 (Sections 4.1, 4.2); Table 4's divergence between mean-pooling and linear-projection on UCF despite similar K600 reconstruction shows sensitivity to this transfer.
  • domain assumption The 'internal dataset' used for interpolation training is representative of natural videos.
    Used in Section 4.4 and Appendix E.1; no description is given, so the interpolation results cannot be assessed externally.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Factorized Video Autoencoders for Efficient Generative Modelling." pith.science (2026). https://pith.science/paper/BJXY2R7B

@misc{pith2026241204452,
  author       = {Pith},
  title        = {Pith review of: Factorized Video Autoencoders for Efficient Generative Modelling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BJXY2R7B}},
  note         = {Machine review of arXiv:2412.04452}
}
read the original abstract

Latent variable generative models have emerged as powerful tools for generative tasks including image and video synthesis. These models are enabled by pretrained autoencoders that map high resolution data into a compressed lower dimensional latent space, where the generative models can subsequently be developed while requiring fewer computational resources. Despite their effectiveness, the direct application of latent variable models to higher dimensional domains such as videos continues to pose challenges for efficient training and inference. In this paper, we propose an autoencoder that projects volumetric data onto a four-plane factorized latent space that grows sublinearly with the input size, making it ideal for higher dimensional data like videos. The design of our factorized model supports straightforward adoption in a number of conditional generation tasks with latent diffusion models (LDMs), such as class-conditional generation, frame prediction, and video interpolation. Our results show that the proposed four-plane latent space retains a rich representation needed for high-fidelity reconstructions despite the heavy compression, while simultaneously enabling LDMs to operate with significant improvements in speed and memory.

Figures

Figures reproduced from arXiv: 2412.04452 by the authors.

Figure 1
Figure 1. Factorized latent representation. Traditional volumet￾ric latents in diffusion models yield a sequence length of t×h×w (top row), which scales linearly with the input size and demands high computational resources. Our proposed factorized represen￾tation reduces sequence length to t×(h+w)+ 2×h×w (bottom row), achieving a more compact latent space that scales sublin￾early with input size, enabling faster, more efficie… view at source ↗
Figure 2
Figure 2. Model overview. The autoencoder first maps the input video into a volumetric latent representation through 3D convolutional architecture, which is then factorized into four planes. Temporal planes are created by mean pooling along the height and width dimensions, capturing time-varying features. Spatial planes are obtained by splitting along the time axis and independently averaging along this dimension, focusing on… view at source ↗
Figure 3
Figure 3. Class-conditional generation results on the UCF dataset. We show every other frame of the 17-frame generated videos from the 128 × 128 models. The temporal continuity and overall frame quality of our factorized model is comparable to the volumetric W.A.L.T. generations. use the Frechet Video Distance (FVD) [ ´ 48] as our primary metric. FVD measures the similarity between the distribu￾tions of generated and real vid… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Interpolation results. We show the 7 interpolated frames for two scenes from the DAVIS-7 [27] dataset, our method generates realistic videos with sharp, detailed frames, achieving quality comparable to VIDIM [27]. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Timing Breakdown. Execution times for the encoder, denoiser, and decoder are reported across varying batch sizes on TPU architectures (v5e and v4) in Rows 1 and 2, and GPU architectures (V100 and A100) in Rows 3 and 4. The comparison includes timings for factorized lat…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 24 canonical work pages

  1. [1]

    Latent-shift: Latent diffu- sion with temporal shift for efficient text-to-video genera- tion

    Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin. Latent-shift: Latent diffu- sion with temporal shift for efficient text-to-video genera- tion. arXiv preprint arXiv:2304.08477, 2023. 1

    arXiv 2023

  2. [2]

    Lumiere: A space- time diffusion model for video generation

    Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Her- rmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, et al. Lumiere: A space- time diffusion model for video generation. arXiv preprint arXiv:2401.12945, 2024. 2

    arXiv 2024

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 1, 2

    arXiv 2023

  4. [4]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22563–22575, 2023. 2

    2023

  5. [5]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), 2023. 1

    work page 2023

  6. [6]

    A short note about kinetics-

    Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-

  7. [7]

    Tensorf: Tensorial radiance fields

    Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. InEuropean con- ference on computer vision, pages 333–350. Springer, 2022. 2

    2022

  8. [8]

    Analog bits: Generating discrete data using diffusion models with self-conditioning

    Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint arXiv:2208.04202, 2022. 5

    arXiv 2022

Show all 65 references
  1. [9]

    3d u-net: learn- ing dense volumetric segmentation from sparse annota- tion

    ¨Ozg¨un C ¸ ic ¸ek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: learn- ing dense volumetric segmentation from sparse annota- tion. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, At...

    2016

  2. [10]

    Emu: Enhanc- ing image generation models using photogenic needles in a haystack

    Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xi- aofang Wang, Abhimanyu Dubey, et al. Emu: Enhanc- ing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807, 2023. 1, 2

    2023 arXiv

  3. [11]

    Ldmvfi: Video frame interpolation with latent diffusion models

    Duolikun Danier, Fan Zhang, and David Bull. Ldmvfi: Video frame interpolation with latent diffusion models. InProceed- ings of the AAAI Conference on Artificial Intelligence, pages 1472–1480, 2024. 2, 6

    2024

  4. [12]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural informa- tion processing systems, 34:8780–8794, 2021. 2

    2021

  5. [13]

    Video frame interpolation: A comprehensive survey

    Jiong Dong, Kaoru Ota, and Mianxiong Dong. Video frame interpolation: A comprehensive survey. ACM Transactions on Multimedia Computing, Communications and Applica- tions, 19(2s):1–31, 2023. 2

    2023

  6. [14]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 2

    2021

  7. [15]

    NVIDIA et. al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025. 2

    2025 arXiv

  8. [16]

    K-planes: Explicit radiance fields in space, time, and appearance

    Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 12479–12488, 2023. 2

    2023

  9. [17]

    Emu video: Factoriz- ing text-to-video generation by explicit image conditioning

    Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Du- val, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factoriz- ing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709, 2023. 1

    2023 arXiv

  10. [18]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 2

    2023 arXiv

  11. [19]

    Photorealistic video generation with diffusion models, 2023

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and Jos ´e Lezama. Photorealistic video generation with diffusion models, 2023. 1, 2, 5, 6, 3

    2023

  12. [20]

    Latent video diffusion models for high-fidelity long video generation

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. 2022. 2

    2022

  13. [21]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 2, 4, 3

    2020

  14. [22]

    Imagen 9 video: High definition video generation with diffusion mod- els

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen 9 video: High definition video generation with diffusion mod- els. arXiv preprint arXiv:2210.02303, 2022. 2

    arXiv

  15. [23]

    Video dif- fusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022. 2, 6

    2022

  16. [24]

    sim- ple diffusion: End-to-end diffusion for high resolution im- ages

    Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. sim- ple diffusion: End-to-end diffusion for high resolution im- ages. In International Conference on Machine Learning , pages 13213–13232. PMLR, 2023. 2

    2023

  17. [25]

    Real-time intermediate flow estimation for video frame interpolation

    Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In European Conference on Com- puter Vision, pages 624–642. Springer, 2022. 6

    2022

  18. [26]

    Scalable adap- tive computation for iterative generation

    Allan Jabri, David Fleet, and Ting Chen. Scalable adap- tive computation for iterative generation. arXiv preprint arXiv:2212.11972, 2022. 6

    2022 arXiv

  19. [27]

    Video inter- polation with diffusion models

    Siddhant Jain, Daniel Watson, Eric Tabellion, Aleksander Hoły´nski, Ben Poole, and Janne Kontkanen. Video inter- polation with diffusion models. In CVPR, 2024. 2, 7, 8

    2024

  20. [28]

    Video interpolation with diffu- sion models

    Siddhant Jain, Daniel Watson, Eric Tabellion, Ben Poole, Janne Kontkanen, et al. Video interpolation with diffu- sion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7341– 7351, 2024. 6

    2024

  21. [29]

    Benchmarking video frame interpolation

    Simon Kiefhaber, Simon Niklaus, Feng Liu, and Simone Schaub-Meyer. Benchmarking video frame interpolation. arXiv preprint arXiv:2403.17128, 2024. 2

    2024 arXiv

  22. [30]

    Hybrid video diffusion models with 2d triplane and 3d wavelet rep- resentation

    Kihong Kim, Haneol Lee, Jihye Park, Seyeon Kim, Kwanghee Lee, Seungryong Kim, and Jaejun Yoo. Hybrid video diffusion models with 2d triplane and 3d wavelet rep- resentation. In European Conference on Computer Vision , pages 148–165. Springer, 2024. 6

    2024

  23. [31]

    Auto-encoding varia- tional bayes

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. arXiv preprint arXiv:1312.6114, 2013. 5

    2013 arXiv

  24. [32]

    Semcity: Semantic scene gener- ation with triplane diffusion

    Jumin Lee, Sebin Lee, Changho Jo, Woobin Im, Juhyeong Seon, and Sung-Eui Yoon. Semcity: Semantic scene gener- ation with triplane diffusion. In CVPR, 2024. 2

    2024

  25. [33]

    Amt: All-pairs multi-field transforms for efficient frame interpolation

    Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun- Le Guo, and Ming-Ming Cheng. Amt: All-pairs multi-field transforms for efficient frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 9801–9810, 2023. 6

    2023

  26. [34]

    Wf-vae: Enhancing video vae by wavelet-driven energy flow for latent video diffusion model

    Zongjian Li, Bin Lin, Yang Ye, Liuhan Chen, Xinhua Cheng, Shenghai Yuan, and Li Yuan. Wf-vae: Enhancing video vae by wavelet-driven energy flow for latent video diffusion model. arXiv preprint arXiv:2411.17459, 2024. 5

    2024 arXiv

  27. [35]

    Open-sora plan: Open-source large video generation model

    Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Li- uhan Chen, et al. Open-sora plan: Open-source large video generation model. arXiv preprint arXiv:2412.00131, 2024. 5

    2024 arXiv

  28. [36]

    Common diffusion noise schedules and sample steps are flawed

    Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 5404–5411, 2024. 5

    2024

  29. [37]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 1, 2

    2023 arXiv

  30. [38]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  31. [39]

    Gener- ating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Gener- ating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019. 2

    2019

  32. [40]

    Film: Frame inter- polation for large motion

    Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless. Film: Frame inter- polation for large motion. In European Conference on Com- puter Vision, pages 250–266. Springer, 2022. 6

    2022

  33. [41]

    High-resolution image syn- thesis with latent diffusion models, 2021, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models, 2021, 2021. 1, 3

    2021

  34. [42]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...

    2015

  35. [43]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  36. [44]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022. 4

    2022 arXiv

  37. [45]

    Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein

    J. Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In CVPR, 2023. 2

    2023

  38. [46]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 3

    2010 arXiv

  39. [47]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 ,

    arXiv

  40. [48]

    Fvd: A new metric for video generation

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Rapha¨el Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019. 5

    2019

  41. [49]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 2

    2017

  42. [50]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neu- ral Information Processing Systems. Curran Associates, Inc.,

  43. [51]

    Phenaki: Variable length video generation from open domain textual descriptions

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kin- dermans, Hernan Moraldo, Han Zhang, Mohammad Taghi 10 Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on...

    2022

  44. [52]

    Omnitokenizer: A joint image-video tokenizer for visual generation

    Junke Wang, Yi Jiang, Zehuan Yuan, BINGYUE PENG, Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint image-video tokenizer for visual generation. In Advances in Neural Information Processing Systems , pages 28281– 28295. Curran Associates, Inc., 2024. 2, 5

    2024

  45. [53]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference...

    2023

  46. [54]

    Sin3dm: Learning a diffusion model from a single 3d tex- tured shape

    Rundi Wu, Ruoshi Liu, Carl V ondrick, and Changxi Zheng. Sin3dm: Learning a diffusion model from a single 3d tex- tured shape. arXiv preprint arXiv:2305.15399, 2023. 2

    2023 arXiv

  47. [55]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 2, 5

    2024 arXiv

  48. [56]

    Magvit: Masked generative video transformer

    Lijun Yu, Yong Cheng, Kihyuk Sohn, Jos ´e Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2023

  49. [57]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, Jos ´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023. 2

    arXiv

  50. [58]

    Language model beats diffusion - tokenizer is key to visual generation

    Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A Ross, and Lu Jiang. Language model beats diffusion - tokenizer is key to vi...

    2024

  51. [59]

    Video probabilistic diffusion models in projected latent space

    Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 18456–18466,

  52. [60]

    Video probabilistic diffusion models in projected latent space

    Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In CVPR, 2023. 1, 6

    2023

  53. [61]

    Efficient video diffusion mod- els via content-frame motion-latent decomposition

    Sihyun Yu, Weili Nie, De-An Huang, Boyi Li, Jinwoo Shin, and Anima Anandkumar. Efficient video diffusion mod- els via content-frame motion-latent decomposition. arXiv preprint arXiv:2403.14148, 2024. 1

    2024 arXiv

  54. [62]

    Cv- vae: A compatible video vae for latent generative video mod- els

    Sijie Zhao, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Muyao Niu, Xiaoyu Li, Wenbo Hu, and Ying Shan. Cv- vae: A compatible video vae for latent generative video mod- els. Advances in Neural Information Processing Systems, 37: 12847–12871, 2025. 2, 5 11 A. Frames vs Reconstructio...

    2025

  55. [65]

    It achieves a reconstruction PSNR of 25.58 and SSIM of 0.717 on videos with 256×256 resolution and 9 temporal frames. E.2. Denoiser We use the same transformer architecture across all three tasks, following the design and hyperparameters outlined in W.A.L.T. [19]. • Class-cond...

  56. [256]

    For the video interpolation task, the autoencoder is 2 trained for 450, 000 iterations with the same batch size of

    The resulting autoencoder achieves a reconstruction performance of 27.11 PSNR and 0.829 SSIM on videos with 128 × 128 resolution and 17 frames. For the video interpolation task, the autoencoder is 2 trained for 450, 000 iterations with the same batch size of

  57. [600]

    arXiv preprint arXiv:1808.01340, 2018. 5

    2018 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.