Pith. sign in

REVIEW 4 major objections 5 minor 25 references

Frozen video foundation models, built for understanding, can be repurposed as the encoder of a video autoencoder, producing latent spaces that yield state-of-the-art class-to-video generation and roughly five-fold faster generative training

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 02:46 UTC pith:4ZM2Y5HC

load-bearing objection A plausible, useful video-representation autoencoder whose central KL-free claim rests on an unmeasured latent-manifold assumption; worth refereeing, but the evidence needs to be much tighter. the 4 major comments →

arxiv 2607.14088 v1 pith:4ZM2Y5HC submitted 2026-07-15 cs.CV

VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders

Zhihao Xie , Junfeng Wu , Xinting Hu , Junchao Huang , Li Jiang This is my paper
classification cs.CV
keywords video representation autoencoderfrozen video foundation modelslatent video generationrepresentation alignmentmulti-codebook quantizationautoregressive video generationdiffusion transformersvideo latent compression
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether the semantic representations from a frozen video foundation model—one trained for understanding, not reconstruction—can be turned into a compact latent space for video generation. The authors answer yes with VideoRAE, which uses the frozen model as a fixed encoder, compresses its multi-scale features with a 1D self-attention projector, and supports both continuous latents for diffusion transformers and discrete multi-codebook tokens for autoregressive models. A representation-alignment loss replaces the usual KL regularization, and the paper reports state-of-the-art gFVDs of 40 (autoregressive) and 93 (diffusion) on UCF-101, with generative training converging about five times faster than with conventional pixel-driven autoencoders. If correct, this means video generation can inherit semantic structure from understanding models instead of learning it from scratch.

Core claim

The paper claims that a frozen, self-supervised video foundation model—trained for understanding, not generation—can serve as the encoder of a video autoencoder. VideoRAE fuses hierarchical features from that frozen model, compresses them with a lightweight 1D self-attention projector into 512 or 1024 tokens, and maps them to either a continuous latent (for diffusion transformers) or discrete multi-codebook tokens (for autoregressive models). A representation-alignment loss, computed by matching decoder features to the frozen teacher's features locally and globally, replaces the KL divergence used in conventional VAEs. On UCF-101 class-to-video generation, the resulting latents give gFVD 40

What carries the argument

The central machinery is the frozen video foundation model used as a fixed semantic encoder, paired with a 1D self-attention projector that compresses its multi-scale features into a compact token sequence. The REPA (representation alignment) loss aligns decoder features with the frozen teacher at local and global scales, which the paper argues regularizes the latent manifold and eliminates the need for KL regularization. For the discrete branch, Multi-Codebook SimVQ splits the latent into K sub-vectors, each quantized against a learnable MLP-mapped frozen base codebook, preserving high dimensionality and avoiding codebook collapse.

Load-bearing premise

The whole speedup and quality gain rests on the claim that the representation-alignment loss alone, without any KL term, keeps the latent space smooth and prior-friendly; the paper offers no direct measurement of that geometry, only downstream metrics, so if alignment does not actually impose that structure, the gains could be caused by the stronger decoder or adversarial loss instead.

What would settle it

Train the same VideoRAE with a conventional KL loss added in place of the REPA loss and compare gFVD and convergence; if the scores stay near 93/40, then REPA's semantic regularization is not the cause. Alternatively, measure the latent-space geometry directly—for example, interpolate between two V-JEPA 2 latents and reconstruct; if interpolations collapse to unrealistic videos, the latent manifold is not as smooth as claimed.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, video generation no longer requires training a pixel-reconstruction encoder from scratch: a frozen understanding model can provide the latent space, simplifying the autoencoder and lowering compute.
  • Generative models trained on these latents converge roughly 5x faster than on conventional video tokenizers, which cuts the training cost of video diffusion and autoregressive models.
  • Because the same frozen encoder supports both continuous and discrete latents, one autoencoder can serve both diffusion-transformer and autoregressive pipelines, unifying the two paradigms.
  • For generation quality, semantic fidelity matters more than pixel fidelity: the model with slightly lower reconstruction (V-JEPA 2-based) yields better gFVD than the one with higher PSNR (VideoMAEv2-based).
  • The approach extends to text-to-video at scale: replacing an existing video VAE with VideoRAE in a 2B-scale T2V system improves VBench scores and convergence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If this generalizes, future video autoencoders could be upgraded by swapping in a better frozen foundation model without redesigning reconstruction, letting representation-learning progress feed directly into generation quality.
  • The paper's observation that a lower-reconstruction latent produces better generation suggests a re-weighting of autoencoder objectives: semantic and structural alignment may be worth trading against PSNR.
  • A testable extension: apply the same frozen-encoder-plus-REPA recipe to other modalities or tasks, such as audio or 3D, where self-supervised models exist but generation still relies on pixel-level autoencoders.
  • The absence of a KL prior means the generator must match the latent distribution induced by the VFM; it would be worth measuring how close that distribution is to a simple prior, and whether the 5x speedup persists when the VFM is changed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces VideoRAE, a video autoencoder that replaces the conventional learnable 3D encoder with a frozen Video Foundation Model (VFM) whose multi-scale hierarchical features are compressed by a lightweight 1D self-attention projector. The resulting latents support both continuous (for DiTs) and discrete (via a Multi-Codebook SimVQ) generative models. A Representation Alignment (REPA) loss aligns decoder features with the frozen VFM teacher and is claimed to replace the KL divergence of standard VAEs. The paper reports strong reconstruction and state-of-the-art class-to-video gFVDs on UCF-101 (40 discrete AR, 93 continuous DiT), about 5× faster convergence than LARP/LTX-VAE baselines, and a controlled 2B-scale text-to-video study showing better VBench scores than LTX-VAE.

Significance. If the results hold, VideoRAE would be a significant step toward unifying video understanding and generation by demonstrating that frozen VFM features can be directly reused as compact, reconstruction-capable, and generation-friendly latent spaces. The paper is well positioned in a timely research direction and includes extensive ablations over quantization, REPA, feature layers, and fusion strategies, which is a strength. However, the central KL-free claim rests on an unmeasured assertion about latent geometry, and the empirical comparisons rely on single-run metrics without code or checkpoints. These limitations currently temper the strength of the contribution, but the core idea is plausible and worthy of further validation.

major comments (4)
  1. [§3.3, Eq. (6)] The paper's core mechanistic claim is that REPA 'implicitly regularizes the geometry of the latent space' and 'completely replaces the traditional KL-divergence constraint.' This is asserted, not demonstrated. The only evidence is downstream gFVD improvements in Table 7, but gFVD is a generation metric that could improve because of better decoder features, GAN losses, or generator capacity, without implying that the latent manifold is actually prior-friendly or topologically regular. Since the KL-free mechanism is a central contribution, the authors should provide direct evidence: e.g., measure the empirical latent distribution against the assumed Gaussian, analyze latent nearest-neighbor/interpolation behavior, or compare against a KL-regularized variant with the same decoder and GAN losses. The current omission is load-bearing for the claim that VideoRAE's convergence speed and SOTA ge
  2. [§4.1, 'Text-to-Video Generation on VBench' and Table 5] The controlled T2V comparison is confounded by a frame-count mismatch: LTX-VAE is trained on 17-frame clips and VideoRAE on 16-frame clips. This changes the token sequence length, temporal receptive field, and effective compute per sample. The paper states the settings are 'comparable,' but the difference makes the comparison not controlled. To support the faster-convergence claim, both models should be trained on the same clip length, or the paper should explicitly show that the one-frame difference is immaterial (e.g., by ablating LTX-VAE on 16 frames). Without this, the VBench advantage and convergence gap cannot be attributed cleanly to the latent space.
  3. [Tables 3–4, Fig. 7] All quantitative claims—especially 'state-of-the-art gFVD' and 'approximately 5× faster convergence'—are based on single runs without variance or significance testing. The differences between some configurations are small (e.g., Table 4: 93 vs 94 for 1024×64 between V-JEPA and VideoMAEv2), and the convergence comparison in Fig. 7 uses different model sizes and token budgets (VideoRAE AR is 1.3B vs LARP 632M). No code or checkpoints are available to reproduce the numbers or verify the 'identical protocols' claim for baselines. This is a significant reproducibility concern that undermines the strength of the central quantitative claims.
  4. [§5 (Conclusion) and overall manuscript] There is no limitations section, and the paper does not acknowledge the absence of direct latent-geometry measurements or the confounds in the convergence comparison. The 'Discussion' paragraph in §4.1 addresses reconstruction vs. generation quality but does not flag these methodological limitations. Given the central claims, the authors should add a dedicated limitations discussion or, better, supply the missing experiments to address the concerns above.
minor comments (5)
  1. [Throughout] The manuscript contains several typos and formatting issues: 'W AN2.1-V AE' should be 'WAN2.1-VAE', 'V AEs' should be 'VAEs', '16×256×256' appears inconsistently, and the title of Table 4 is missing a closing parenthesis. These are cosmetic but should be cleaned.
  2. [§3.1, Eq. (2)] The 1D Projector is described only as 'based on self-attention mechanisms.' Please specify the number of layers, heads, and dimension, as this module is a key component for the compression and its design choices affect the latent capacity.
  3. [§4, 'Implementation details'] Hyperparameters such as the REPA loss weight λ (Eq. 6) and λ_repa (Eq. 7) are not reported. The paper also does not state which V-JEPA 2 / VideoMAEv2 model version is used. Please provide these details for reproducibility.
  4. [References] Several references carry future-dated arXiv IDs (e.g., '2603.14482', '2601.03233', '2506.09985' with 2025 date), which is unusual and should be checked for correctness.
  5. [Tables 3 and 4] The evaluation protocols use different numbers of generated samples (10K for discrete AR, 2K for continuous DiT). This makes cross-paradigm gFVD comparisons difficult and should be noted explicitly.

Circularity Check

0 steps flagged

No circular derivation: generation results are benchmark-tested; the REPA latent-manifold claim is an unmeasured assertion, and the only self-citation is non-load-bearing.

full rationale

The paper's derivation chain is self-contained: VideoRAE's latents are produced by a frozen VFM encoder plus a learned 1D projector, and the fitted objectives are pixel reconstruction, GAN, VQ, and REPA losses. Downstream gFVD and VBench scores are independent benchmarks, not quantities built into those losses, so the headline generation results are not predictions forced by construction. The REPA objective (Eq. 6) does use the same frozen VFM as both feature source and teacher, which makes 'semantic preservation' a form of feature matching to the encoder's own representation; this is a training-loss design choice rather than a derivation that equates the claim with the input. The stronger assertion that REPA 'implicitly regularizes the geometry of the latent space' and 'completely replaces the traditional KL-divergence constraint' (Sec. 3.3) is not demonstrated by direct latent-geometry measurements, but that is a validation gap and correctness risk, not circularity. The only self-citation found is UniTok (Ma et al., 2025), which includes coauthor Junfeng Wu; it appears in Sec. 2.3 as a related-work example and is not load-bearing. No uniqueness theorem is imported from the authors' prior work, no fitted parameter is renamed as a prediction, and no ansatz is smuggled in via citation. Score 1 reflects the minor non-load-bearing self-citation and the unmeasured latent-manifold assertion, not circular reasoning.

Axiom & Free-Parameter Ledger

7 free parameters · 3 axioms · 0 invented entities

The central claim rests on the assumptions that frozen VFM features can be decoded to pixels after compression and that REPA is a sufficient prior replacement. Both are empirical postulates; no formal bounds or direct latent-geometry measurements are provided. Hyperparameters such as lambda, lambda_repa, token counts, codebook size, and layer set are hand/ablation-chosen; they are not fitted to a target metric but do enter the method.

free parameters (7)
  • REPA global-loss weight lambda
    Eq. 6 weights the global alignment term; the chosen value is not reported and no sensitivity study is shown.
  • REPA loss weight lambda_repa
    Eq. 7 multiplies L_REPA; value not reported.
  • Number of latent tokens N_latent = 512 or 1024
    Encoder compresses to 512/1024 tokens depending on variant; chosen as a config, ablations only compare 512 vs 1024 in some cases.
  • Latent channel dimension = 32/64 continuous; 512 discrete (4x128)
    Continuous latent linear projection to 32/64 dims; discrete uses 4 sub-codebooks of 128 dim each; hand-set.
  • Number of sub-codebooks K and vocabulary V = K=4, V=4096 each
    Multi-Codebook SimVQ hyperparameters; no search reported, only comparison across quantization strategies.
  • Layer set S for VFM features = Layers 8-24
    Selected by ablation (Table 8); other layer sets are choices.
  • Number of decoder learnable 3D tokens = 1024
    Decoder concatenates 1024 learnable 3D tokens; set without ablation.
axioms (3)
  • domain assumption Frozen VFM features (V-JEPA2/VideoMAEv2) contain sufficient information to reconstruct 16x256x256 RGB videos after 1D compression.
    The entire method assumes the VFM encoder's multi-layer features are a lossless-enough representation; no analysis of information content or compression bounds is given.
  • domain assumption REPA loss (Eqs. 4-6) enforces a well-behaved latent manifold sufficient to replace the KL prior.
    Sec. 3.3 claims semantic alignment 'implicitly regularizes the geometry' without measuring the latent distribution; central to KL-free training.
  • domain assumption Backbone/generator training recipes are equivalent across compared autoencoders (e.g., DiT on LTX-VAE vs VideoRAE) except for the autoencoder.
    Comparisons assume all other factors are held fixed; the 17-frame vs 16-frame T2V comparison is an explicit violation.

pith-pipeline@v1.3.0-alltime-deepseek · 15014 in / 10367 out tokens · 99364 ms · 2026-08-02T02:46:41.816273+00:00 · methodology

0 comments
read the original abstract

Video generative models commonly rely on latent spaces learned by 3D Variational Autoencoders (3D-VAEs). However, conventional 3D-VAEs are mainly optimized for pixel-level reconstruction, which can limit the semantic and spatio-temporal structure captured by their latents. Meanwhile, Video Foundation Models (VFMs) such as V-JEPA 2 and VideoMAEv2 show strong video understanding capabilities, yet whether their frozen representations can be transformed into compact, reconstruction-capable, and generation-friendly video latents remains largely unexplored. We answer this question with VideoRAE, a representation autoencoder that leverages multi-scale hierarchical features from a frozen video foundation encoder and compresses them with a lightweight 1D self-attention projector. VideoRAE supports both continuous latents for Diffusion Transformers and discrete tokens for autoregressive models via multi-codebook high-dimensional quantization. During decoding, a local-and-global representation alignment objective with the frozen VFM teacher improves semantic preservation and enables training without KL regularization. Experiments show that VideoRAE achieves strong reconstruction in both continuous and discrete regimes. On UCF-101, it obtains state-of-the-art class-to-video gFVDs of 40 and 93 with AR and DiT generators, respectively, while converging approximately 5x faster than competing autoencoder baselines. In a controlled 2B-scale text-to-video study, replacing LTX-VAE with VideoRAE leads to faster convergence under comparable settings. These results validate frozen VFM representations as versatile and generation-friendly video latents. The model and code will be released on https://zhxie0117.github.io/VideoRAE.

Figures

Figures reproduced from arXiv: 2607.14088 by Junchao Huang, Junfeng Wu, Li Jiang, Xinting Hu, Zhihao Xie.

Figure 1
Figure 1. Figure 1: Conceptual comparison between traditional video tokenizers and VideoRAE. Traditional [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overall architecture of VideoRAE. Given an input video, multi-scale hierarchical fea￾tures are first extracted from a frozen VFM and fused, followed by a 1D self-attention projector that dynamically compresses them into compact base tokens. These tokens are then formatted into either a continuous latent space via linear projection for Diffusion Transformers, or a discrete latent space via Multi-Codebook Si… view at source ↗
Figure 3
Figure 3. Figure 3: Visualization comparison of the Discrete-token reconstruction on the UCF-101 dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Visualization comparison of the Continuous-space reconstruction on the UCF-101 dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of the Discrete-token generation results on the UCF-101 dataset. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of the Continuous-space generation results on the UCF-101 dataset. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Training convergence speed comparison in AR (left) and DiT (right) generative modeling. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Convergence comparison between VideoRAE and LTX-VAE on VBench. [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 19 linked inside Pith

  1. [1]

    Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575,

    Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chat- topadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575,

    Pith/arXiv arXiv

  2. [3]

    A short note about kinetics-600.arXiv preprint arXiv:1808.01340,

    Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600.arXiv preprint arXiv:1808.01340,

    Pith/arXiv arXiv

  3. [6]

    Ltx-video: Realtime video latent diffusion

    Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.00103,

    Pith/arXiv arXiv

  4. [9]

    Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization.arXiv preprint arXiv:2402.03161,

    Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et al. Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization.arXiv preprint arXiv:2402.03161,

    arXiv

  5. [10]

    Unitok: A unified tokenizer for visual generation and understanding.arXiv preprint arXiv:2502.20321,

    Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiao- juan Qi. Unitok: A unified tokenizer for visual generation and understanding.arXiv preprint arXiv:2502.20321,

    arXiv

  6. [11]

    Latte: Latent diffusion transformer for video generation.arXiv preprint arXiv:2401.03048,

    Xin Ma, Yaohui Wang, Xinyuan Chen, Gengyun Jia, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation.arXiv preprint arXiv:2401.03048,

    Pith/arXiv arXiv

  7. [12]

    V-jepa 2.1: Unlocking dense features in video self-supervised learning.arXiv preprint arXiv:2603.14482,

    Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. V-jepa 2.1: Unlocking dense features in video self-supervised learning.arXiv preprint arXiv:2603.14482,

    Pith/arXiv arXiv

  8. [13]

    Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193,

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193,

    Pith/arXiv arXiv

  9. [16]

    Ucf101: A dataset of 101 human actions classes from videos in the wild.arXiv preprint arXiv:1212.0402,

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild.arXiv preprint arXiv:1212.0402,

    Pith/arXiv arXiv

  10. [18]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786,

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786,

    Pith/arXiv arXiv

  11. [20]

    Larp: Tokenizing videos with a learned autoregressive generative prior.arXiv preprint arXiv:2410.21264, 2024a

    Hanyu Wang, Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava. Larp: Tokenizing videos with a learned autoregressive generative prior.arXiv preprint arXiv:2410.21264, 2024a. Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint image-video tokenizer for visual generation.Advances in Neural Informati...

    Pith/arXiv arXiv

  12. [21]

    Internvideo: General video foundation models via generative and discriminative learning.arXiv preprint arXiv:2212.03191,

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning.arXiv preprint arXiv:2212.03191,

    Pith/arXiv arXiv

  13. [22]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. InInternational Conference on Learning Representations, volume 2025, pp. 83048–83077,

    2025

  14. [23]

    Deco-vae: Learning compact latents for video reconstruction via decoupled repre- sentation.arXiv preprint arXiv:2511.14530,

    Xiangchen Yin, Jiahui Yuan, Zhangchi Hu, Wenzhang Sun, Jie Chen, Xiaozhen Qiao, Hao Li, and Xiaoyan Sun. Deco-vae: Learning compact latents for video reconstruction via decoupled repre- sentation.arXiv preprint arXiv:2511.14530,

    arXiv

  15. [24]

    Videorepa: Learning physics for video generation through relational alignment with foundation models.arXiv preprint arXiv:2505.23656,

    Xiangdong Zhang, Jiaqi Liao, Shaofeng Zhang, Fanqing Meng, Xiangpeng Wan, Junchi Yan, and Yu Cheng. Videorepa: Learning physics for video generation through relational alignment with foundation models.arXiv preprint arXiv:2505.23656,

    Pith/arXiv arXiv

  16. [25]

    Vision foundation models as effective visual tokenizers for autoregressive image generation.arXiv preprint arXiv:2507.08441, 2025a

    Anlin Zheng, Xin Wen, Xuanyang Zhang, Chuofan Ma, Tiancai Wang, Gang Yu, Xiangyu Zhang, and Xiaojuan Qi. Vision foundation models as effective visual tokenizers for autoregressive image generation.arXiv preprint arXiv:2507.08441, 2025a. Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with repre- sentation autoencoders.arXiv...

    arXiv

  17. [2012]

    Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525,

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525,

    Pith/arXiv arXiv

  18. [2014]

    Dera: Decoupled representation alignment for video tokenization.arXiv preprint arXiv:2512.04483,

    Pengbo Guo, Junke Wang, Zhen Xing, Chengxu Liu, Daoguo Dong, Xueming Qian, and Zux- uan Wu. Dera: Decoupled representation alignment for video tokenization.arXiv preprint arXiv:2512.04483,

    arXiv

  19. [2017]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314,

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314,

    Pith/arXiv arXiv

  20. [2021]

    Long video generation with time-agnostic vqgan and time-sensitive transformer

    Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. arXiv preprint arXiv:2204.03638,

    Pith/arXiv arXiv

  21. [2022]

    Dinov3.arXiv preprint arXiv:2508.10104,

    Oriane Sim´eoni, Huy V V o, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Micha ¨el Ramamonjisoa, et al. Dinov3.arXiv preprint arXiv:2508.10104,

    Pith/arXiv arXiv

  22. [2023]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952,

    15 Preprint Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952,

    Pith/arXiv arXiv

  23. [2024]

    Ltx-2: Efficient joint audio-visual foundation model.arXiv preprint arXiv:2601.03233,

    Yoav HaCohen, Benny Brazowski, Nisan Chiprut, Yaki Bitterman, Andrew Kvochko, Avishai Berkowitz, Daniel Shalem, Daphna Lifschitz, Dudu Moshe, Eitan Porat, et al. Ltx-2: Efficient joint audio-visual foundation model.arXiv preprint arXiv:2601.03233,

    Pith/arXiv arXiv

  24. [2025]

    V-jepa 2: Self-supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985,

    Mahmoud Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Ar- naud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li,...

    Pith/arXiv arXiv

  25. [2026]

    Cogvideo: Large-scale pre- training for text-to-video generation via transformers.arXiv preprint arXiv:2205.15868,

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers.arXiv preprint arXiv:2205.15868,

    Pith/arXiv arXiv