Pith. sign in

REVIEW 3 major objections 5 minor 51 references

RefTok: Reference-Based Tokenization for Video Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By keeping one reference frame unquantized, RefTok reconstructs text, faces, and fine patterns that reference-less tokenizers lose, improving reconstruction metrics by 36.7% at equal compression.

desk verdict RefTok's reference-bypass idea is novel and the qualitative gains are real, but the headline compression ratio ignores the cost of the unquantized reference side channel. read the letter →

arxiv 2507.02862 v1 pith:X2VGL66Y submitted 2025-07-03 cs.CV

Xiang Fan , Xiaohang Sun , Kushan Thakkar , Zhu Liu , Vimal Bhat , Ranjay Krishna , Xiang Hao This is my paper · ORCID
classification cs.CV
keywords videotokenizationreference-basedcompressionquantizationbypassgenerationtemporalredundancyconditionalreconstructiondiscretelatentspace
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RefTok is a video tokenizer that keeps one reference frame unquantized and encodes every other frame relative to it. The paper's claim is that this single design change lets a neural tokenizer reconstruct a larger range of RGB content, such as legible text, stable faces, and small patterns, than reference-less tokenizers at the same or higher compression. Across four video datasets, RefTok reports average gains of 36.7% across PSNR, SSIM, and LPIPS at a 1024:1 compression ratio, and a generation model trained on its latents outperforms a baseline with four times more parameters by an average of 27.9%. The point is to reopen a design dimension, reference frames, that learned tokenizers had dropped, bringing neural video tokenization closer to classical codecs. If this holds, discrete video tokenization can be made more accurate without sacrificing compression.

What carries the argument

The central mechanism is the quantization bypass: an unquantized reference frame's encoder representation $h_r$ is fed directly to the decoder as a conditioning signal, while target frames' tokens are quantized to $z_t$ and decoded as $\hat{x}_t = D(z_t | h_r)$. A one-way attention mask in a masked-autoencoder-style transformer keeps target information from leaking into the reference representation, preserving causality. Techniques such as training at higher frame intervals and codebook splitting guard against reference posterior collapse, where the decoder would simply copy the reference frame and ignore the latent tokens.

What would settle it

Recompute rate-distortion with the reference frame's compressed bit cost included, for example by compressing the first frame with an intra-frame image codec and adding its bytes to the target tokens' budget, or evaluate with reference frames that carry little useful information such as static or heavily corrupted frames; if the PSNR, SSIM, and LPIPS gains over the baselines vanish or reverse, the compression-equivalence premise fails.

Watch

Extended reading notes

Core claim

On its own terms, RefTok's central discovery is that the bottleneck layer need not quantize everything. By passing continuous reference-frame features straight to the decoder and quantizing only the target tokens under a one-way attention barrier, the encoder has to represent only the temporal difference between reference and target, and the decoder can consult an information-rich continuous reference when reconstructing. This shifts the hard part of video compression from inventing all pixels from a small codebook to encoding what changed, which is why RefTok preserves text, faces, and fine patterns that reference-less tokenizers distort. The paper positions this as a Pareto improvement: better reconstruction at the same or higher compression ratio, with faster and more memory-efficient inference.

Load-bearing premise

The headline 1024:1 compression ratio does not include the bits needed to store or transmit the unquantized reference frame, and if those bits are counted, or if the reference is not a clean, informative first frame, the reported gains shrink.

Editorial extensions

If this is right

  • At the same 1024:1 compression, RefTok reconstructs text, faces, and repetitive patterns from the reference frame, where reference-less tokenizers blur or distort them.
  • The reported 36.7% average improvement across PSNR, SSIM, and LPIPS means reference-based conditioning can push the Pareto front of discrete video tokenization instead of trading quality against compression.
  • Downstream video-generation models trained on RefTok latents can match or beat a model with roughly 4 times more parameters, so better tokenization directly improves generation quality without a larger generator.
  • Because RefTok keeps the encoder and decoder architecture and only changes how frames are routed around the bottleneck, existing tokenizers could adopt the reference-bypass design at a low compute cost.
  • Conditional applications such as zero-shot video editing become single-forward-pass operations by swapping in an edited reference frame at decoding time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reference frame's bits were counted in the compression ratio, for instance by compressing the first frame with a strong image codec and adding those bytes to the token budget, the effective rate would be higher than 1024:1, so the honest rate-distortion gain remains an open question.
  • The method is most natural when a high-quality reference frame already exists, such as keyframe-based coding or image-to-video generation; for unconditional generation, the model must first produce an anchor frame, so the gains may shift downstream.
  • The reference-posterior-collapse fixes suggest reference frames should be informative but not too similar to the targets, making adaptive reference-frame selection among candidate frames a direct and testable extension.
  • If skipping quantization is the source of the gains, then codebook size and quality matter less for video than previously thought; the bottleneck's role becomes representing residuals rather than full frames.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. RefTok proposes a reference-based video tokenizer in which an unquantized reference frame embedding bypasses the quantization bottleneck and conditions the decoder, while all other frames are quantized as usual. The paper claims that this design improves reconstruction PSNR/SSIM/LPIPS by an average of 36.7% over Cosmos and MAGVIT at the same or higher compression ratios across K600, UCF-101, BAIR, and DAVIS, and that a generation model trained on RefTok tokens outperforms both MAGVIT-B and MAGVIT-L on BAIR by an average of 27.9%. The method is evaluated with a fixed first-frame reference, along with ablations, speed/memory measurements, and a zero-shot video-editing application.

Significance. If the quantitative claims were fully supported, RefTok would be a meaningful step for conditional video generation and editing: the idea of letting reference frames skip quantization is simple, plausible, and well-motivated by temporal redundancy, and the qualitative examples of text, face, and fine-pattern preservation are compelling. The paper also evaluates on four datasets, reports speed and memory efficiency, and demonstrates a practical editing use case. However, the central significance depends on the compression-ratio accounting and on whether baselines receive matching conditioning; as presented, the quantitative gains may largely reflect uncharged reference information rather than tokenizer quality. The contribution is best viewed as a conditional tokenization design whose unconditional compression claims are not yet established.

major comments (3)
  1. [§4.4, Eq. (3), Fig. 4] The headline 1024:1 compression ratio does not account for the reference frame. In Eq. (3), the reference embedding hr = E(xr) bypasses the quantizer and is passed to the decoder, and §4.4 states that the first frame is used as the reference. This is an unquantized, effectively high-rate representation of a ground-truth frame, and no bits are charged for it in the reported ratio. During reconstruction of a 16-frame clip, the reference frame is a substantial side channel that reference-less baselines (Cosmos, MAGVIT, etc.) do not receive. Consequently, the Pareto plot in Figure 4 and the average 36.7% improvement in Table 1 conflate free conditioning information with architectural or tokenizer gains. Please provide rate-distortion comparisons that include the reference-frame bit cost (e.g., by quantizing or otherwise coding the reference and adding its bitrate), or explicitly restrict all claims to conditional reconstruction where the reference is a free input and remove the unconditional 'same compression ratio' wording.
  2. [§4.5, Table 2] The generation comparison does not establish that baselines receive the same conditioning. RefTok's tokens are produced with ground-truth reference conditioning, and the generation model is trained on these tokens; the MAGVIT-B/L numbers in Table 2 are reported without any stated reference-conditioning protocol. If the MAGVIT generators are not conditioned on the first frame at inference, the reported 27.9% average improvement may reflect extra information supplied to RefTok rather than a fair tokenizer comparison. Please specify the exact conditioning protocol for every method, including whether the reference (first) frame is provided to the baseline generators, and report results under matched conditioning wherever possible.
  3. [Tables 1–3] All quantitative results appear to be single runs with no error bars, confidence intervals, or significance tests. The abstract's claim of 'significantly outperforms' and the averages of 36.7% and 27.9% are therefore not statistically supported, particularly for FVD where run-to-run variation can be large. Please state the number of seeds, report means with variance (or confidence intervals), and indicate which differences are statistically significant.
minor comments (5)
  1. [§1] There is a typo in 'MAGVIG-B-sized'; it should read 'MAGVIT-B-sized'.
  2. [§2] The phrase 'latent-space probabilitics models' contains a typo; 'probabilistic' is intended.
  3. [Fig. 4] The compression-ratio axis appears to be logarithmic but is not labeled as such, and several legend labels are difficult to read; please improve the figure clarity.
  4. [§3.3 / Training details] The paper refers to an appendix for training details, but no appendix is included in the submitted manuscript; please include the appendix or provide sufficient details for reproduction (e.g., exact ViTMAE initialization, 2D-to-3D position-embedding inflation, and fine-tuning schedule).
  5. [Table 3] The ablation on the number of reference frames reports only L1 error with no variance; adding more reference frames slightly degrades performance, which is not discussed. A brief explanation or additional qualitative comparison would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity established: RefTok is an empirical conditional-tokenization system, and its reference-frame conditioning is a benchmarking caveat, not a derivation that reduces to its inputs.

full rationale

RefTok is an empirical systems paper. Its central claims are measured reconstruction and generation quality, not first-principles derivations, so most circularity patterns do not apply. The architecture in Eq. (3) conditions the decoder on the unquantized reference embedding h_r = E(x_r), and Section 4.4 states that the first frame is used as the reference frame while maintaining a 1024:1 compression ratio. This means the reported compression ratio does not charge bits for the reference frame, and the comparison with reference-less baselines is conditional on free side information. That is a benchmarking and accounting caveat, not circular reasoning: the model is explicitly designed to consume the reference, the reconstruction loss is computed only on target tokens, and the superiority claim is empirical rather than derived from the conditioning itself. No parameter is fitted to a subset of data and then reported as a prediction of a closely related quantity, no load-bearing argument reduces to a self-citation, and no uniqueness theorem is imported from the authors' prior work. The MAE initialization and quantizer choices are standard external components, and the paper's own limitation section acknowledges that a reference frame is required. Therefore no circular step is established, and the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on temporal redundancy between reference and target, plus an accounting premise that reference tokens are free. The accounting premise is the least supported.

free parameters (2)
  • Patch size = [4, 16, 16]
    Chosen by hand; sets token count and compression ratio; ablation shows [1, 32, 32] performs worse (Table 3).
  • Reference frame selection = First frame of each 16-frame clip
    Chosen by hand; paper acknowledges reference selection is not always clear and lists adaptive selection as future work (Section 5).
assumptions (3)
  • domain assumption Reference and target frames share high mutual information
    Section 3 states 'We posit that xr and xt share a high amount of mutual information'; the method is built on this temporal redundancy.
  • domain assumption A pre-trained ViTMAE encoder's reference representation is detailed enough for near-perfect reconstruction
    Introduction insight (1) assumes the MAE representation contains enough information to reconstruct the original frame, justifying the quantization bypass.
  • domain assumption The one-way attention mask prevents target-to-reference information leakage
    Section 3.3 relies on this to preserve causality; if leakage occurs, reconstruction quality gains become trivial.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RefTok: Reference-Based Tokenization for Video Generation." pith.science (2026). https://pith.science/paper/X2VGL66Y

@misc{pith2026250702862,
  author       = {Pith},
  title        = {Pith review of: RefTok: Reference-Based Tokenization for Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X2VGL66Y}},
  note         = {Machine review of arXiv:2507.02862}
}
read the original abstract

Effectively handling temporal redundancy remains a key challenge in learning video models. Prevailing approaches often treat each set of frames independently, failing to effectively capture the temporal dependencies and redundancies inherent in videos. To address this limitation, we introduce RefTok, a novel reference-based tokenization method capable of capturing complex temporal dynamics and contextual information. Our method encodes and decodes sets of frames conditioned on an unquantized reference frame. When decoded, RefTok preserves the continuity of motion and the appearance of objects across frames. For example, RefTok retains facial details despite head motion, reconstructs text correctly, preserves small patterns, and maintains the legibility of handwriting from the context. Across 4 video datasets (K600, UCF-101, BAIR Robot Pushing, and DAVIS), RefTok significantly outperforms current state-of-the-art tokenizers (Cosmos and MAGVIT) and improves all evaluated metrics (PSNR, SSIM, LPIPS) by an average of 36.7% at the same or higher compression ratios. When a video generation model is trained using RefTok's latents on the BAIR Robot Pushing task, the generations not only outperform MAGVIT-B but the larger MAGVIT-L, which has 4x more parameters, across all generation metrics by an average of 27.9%.

Figures

Figures reproduced from arXiv: 2507.02862 by the authors.

Figure 1
Figure 1. REFTOK is a reference-based tokenizer: instead of treating each block of frames independently (shown in box (a)), REFTOK allows encoding and decoding the frames conditioned on one or more reference frames (box (b)). At reconstruction time, traditional tokenizers take no reference frames while REFTOK conditions on reference frames (top half of (c)). When the tokens are decoded, REFTOK preserves the appearance of vari… view at source ↗
Figure 2
Figure 2. REFTOK encodes the target frames (blue) conditioned on reference frames (green), and allows the encoded reference frames to bypass the bottleneck layer as a conditioning signal for the decoder, allowing for higher-quality, more detailed and coherent reconstructions. However, most video frames have a high degree of tem￾poral redundancy with the reference frame. REFTOK lever￾ages this redundancy to improve the quality… view at source ↗
Figure 3
Figure 3. Reconstruction results of REFTOK. Our method is capable of reconstructing a wide variety of concepts that are present in the reference frame, including text, human faces, detailed objects, and small patterns, even with movements and angle changes. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of compression ratio vs. recons. quality on [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Results from a downstream video generation model trained on the BAIR Robot Pushing dataset with R [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: REFTOK can perform simple video editing, such as changing the shape and color of a life vest, modifying the color of a flamingo and SUV car, and adding wings to a swan, without run￾ning expensive video diffusion models [14] or atlas mapping [19]. This is achieved by us…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 46 canonical work pages

  1. [1]

    Towards high resolution video generation with progressive growing of sliced wasserstein gans, 2018

    Dinesh Acharya, Zhiwu Huang, Danda Pani Paudel, and Luc Van Gool. Towards high resolution video generation with progressive growing of sliced wasserstein gans, 2018. 3

    work page 2018

  2. [2]

    Fitvid: Overfitting in pixel-level video prediction, 2021

    Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. Fitvid: Overfitting in pixel-level video prediction, 2021. 3

    work page 2021

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023. 3

    work page 2023

  4. [4]

    Align your latents: High-resolution video synthesis with la- tent diffusion models, 2023

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models, 2023. 3

    work page 2023

  5. [5]

    Efros, and Tero Karras

    Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yu Liu, Alexei A. Efros, and Tero Karras. Generating long videos of dynamic scenes, 2022. 3

    work page 2022

  6. [6]

    A short note about kinetics- 600, 2018

    Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics- 600, 2018. 5

    work page 2018

  7. [7]

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer, 2022. 3

    work page 2022

  8. [8]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024. 3

    work page 2024

Show all 51 references
  1. [9]

    Adversar- ial video generation on complex datasets, 2019

    Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversar- ial video generation on complex datasets, 2019. 3

    2019

  2. [10]

    Av1 bitstream & decod- ing process specification

    Peter de Rivaz and Jack Haughton. Av1 bitstream & decod- ing process specification. 2018. 2

    2018

  3. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  4. [12]

    Lee, and Sergey Levine

    Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections, 2017. 5

    2017

  5. [13]

    Taming transformers for high-resolution image synthesis, 2021

    Patrick Esser, Robin Rombach, and Bj ¨orn Ommer. Taming transformers for high-resolution image synthesis, 2021. 2

    2021

  6. [14]

    Videoshop: Localized semantic video editing with noise-extrapolated diffusion inversion, 2024

    Xiang Fan, Anand Bhattad, and Ranjay Krishna. Videoshop: Localized semantic video editing with noise-extrapolated diffusion inversion, 2024. 8

    2024

  7. [15]

    Rv-gan: Re- current gan for unconditional video generation

    Sonam Gupta, Arti Keshari, and Sukhendu Das. Rv-gan: Re- current gan for unconditional video generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 2024–2033,

    2024

  8. [16]

    Masked autoencoders are scalable vision learners, 2021

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021. 2, 4, 5

    2021

  9. [17]

    Advanced video coding for generic audiovisual services

    Itu-T and Iso Iec Jtc. Advanced video coding for generic audiovisual services. 2010. 2, 3

    2010

  10. [18]

    Rehg, and Pinar Yanardag

    Ozgur Kara, Bariscan Kurtkaya, Hidir Yesiltepe, James M. Rehg, and Pinar Yanardag. Rave: Randomized noise shuf- fling for fast and consistent video editing with diffusion mod- els, 2023. 8

    2023

  11. [19]

    Lay- ered neural atlases for consistent video editing, 2021

    Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. Lay- ered neural atlases for consistent video editing, 2021. 8

    2021

  12. [20]

    Auto-encoding varia- tional bayes, 2022

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes, 2022. 2

    2022

  13. [21]

    Ross, Bryan Seybold, and Lu Jiang

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos ´e Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vigh- nesh Birodkar, Jimmy Yan, Ming-Chang Chiu, Krishna So- mandepalli, Hassan Akbari, Yair Alon, Yong Cheng, Josh Dillon, Agrim Gupta, Meera Hahn, Anja Hauth, David Hen- don, ...

    2024

  14. [22]

    Autoregressive image generation without vec- tor quantization, 2024

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vec- tor quantization, 2024. 3

    2024

  15. [23]

    An algorithm for vector quantizer design

    Yoseph Linde, Andres Buzo, and Robert Gray. An algorithm for vector quantizer design. IEEE Transactions on Commu- nications, 28(1):84–95, 1980. 5

    1980

  16. [24]

    Snap video: Scaled spatiotemporal transformers for text-to-video synthesis, 2024

    Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, and Sergey Tulyakov. Snap video: Scaled spatiotemporal transformers for text-to-video synthesis, 2024. 3

    2024

  17. [25]

    Finite scalar quantization: Vq-vae made simple, 2023

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple, 2023. 5

    2023

  18. [26]

    Hotshot-XL, 2023

    John Mullan, Duncan Crawbuck, and Aakash Sastry. Hotshot-XL, 2023. 3

    2023

  19. [27]

    Perazzi, J

    F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In Computer Vision and Pattern Recognition, 2016. 5

    2016

  20. [28]

    Fatezero: Fus- ing attentions for zero-shot text-based video editing, 2023

    Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fus- ing attentions for zero-shot text-based video editing, 2023. 8

    2023

  21. [29]

    Cosmos tok- enizer: A suite of image and video neural tokenizers, 2024

    Fitsum Red, Jinwei Gu, Xian Liu, Songwei Ge, Ting-Chun Wang, Haoxiang Wang, and Ming-Yu Liu. Cosmos tok- enizer: A suite of image and video neural tokenizers, 2024. 2, 3, 5, 7

    2024

  22. [30]

    High-resolution image syn- thesis with latent diffusion models, 2022

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models, 2022. 2

    2022

  23. [31]

    Tempo- ral generative adversarial nets with singular value clipping,

    Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Tempo- ral generative adversarial nets with singular value clipping,

  24. [32]

    9 Make-a-video: Text-to-video generation without text-video data, 2022

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. 9 Make-a-video: Text-to-video generation without text-video data, 2022. 3

    2022

  25. [33]

    Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2, 2022

    Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elho- seiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2, 2022. 3

    2022

  26. [34]

    Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012. 5

    2012

  27. [35]

    Diffusion model-based video editing: A survey, 2024

    Wenhao Sun, Rong-Cheng Tu, Jingyi Liao, and Dacheng Tao. Diffusion model-based video editing: A survey, 2024. 8

    2024

  28. [36]

    Mocogan: Decomposing motion and content for video generation, 2017

    Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation, 2017. 3

    2017

  29. [37]

    Neural discrete representation learning,

    Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning,

  30. [38]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. 4

    2023

  31. [39]

    Phenaki: Variable length video generation from open domain textual description, 2022

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kin- dermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description, 2022. 2, 3

    2022

  32. [40]

    Larp: Tokenizing videos with a learned autoregressive generative prior, 2024

    Hanyu Wang, Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava. Larp: Tokenizing videos with a learned autoregressive generative prior, 2024. 3

    2024

  33. [41]

    Modelscope text-to-video technical report, 2023

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report, 2023. 3

    2023

  34. [42]

    Omnitokenizer: A joint image- video tokenizer for visual generation, 2024

    Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint image- video tokenizer for visual generation, 2024. 2, 7

    2024

  35. [43]

    Videocomposer: Compositional video synthesis with motion controllability, 2023

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Ji- uniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jin- gren Zhou. Videocomposer: Compositional video synthesis with motion controllability, 2023. 3

    2023

  36. [44]

    Gmflow: Learning optical flow via global matching, 2022

    Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. Gmflow: Learning optical flow via global matching, 2022. 8

    2022

  37. [45]

    Videogpt: Video generation using vq-vae and trans- formers, 2021

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and trans- formers, 2021. 7

    2021

  38. [46]

    Elastictok: Adaptive tok- enization for image and video, 2024

    Wilson Yan, Matei Zaharia, V olodymyr Mnih, Pieter Abbeel, Aleksandra Faust, and Hao Liu. Elastictok: Adaptive tok- enization for image and video, 2024. 2, 5

    2024

  39. [47]

    Cogvideox: Text-to-video diffusion models with an expert transformer, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Xiaotao Gu, Yuxuan Zhang, Weihan Wang, Yean Cheng, Ting Liu, Bin Xu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models ...

    2024

  40. [48]

    Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang

    Lijun Yu, Yong Cheng, Kihyuk Sohn, Jos ´e Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang. Magvit: Masked generative video transformer, 2023. 2, 3, 7

    2023

  41. [49]

    Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vigh- nesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G

    Lijun Yu, Jos ´e Lezama, Nitesh B. Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vigh- nesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats dif- fusion – ...

    2024

  42. [50]

    Generating videos with dynamics-aware implicit generative adversarial net- works, 2022

    Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial net- works, 2022. 3

    2022

  43. [51]

    Show-1: Marrying pixel and latent diffusion models for text-to-video generation, 2023

    David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation, 2023. 3 10

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.