V-RAE: Rethinking Video Latent Spaces for Generation

Hao Fei; Minghui Guo; Shengqiong Wu

REVIEW 4 major objections 5 minor 32 references

V-RAE: Rethinking Video Latent Spaces for Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Frozen vision foundation model features, compacted by a temporal pooler and decoded by a spatiotemporal Transformer, form a video latent space that reconstructs competitively with large pretrained video VAEs, generates more faithfully…

desk verdict A credible extension of RAE to video with strong semantic-preservation results, but the headline generation gains are confounded by asymmetric generator components; worth refereeing with controls. read the letter →

arxiv 2608.13556 v1 pith:ZQJQWUFN submitted 2026-08-13 cs.CV

Minghui Guo , Shengqiong Wu , Hao Fei This is my paper · ORCID

classification cs.CV

keywords videorepresentationautoencoderfrozenvisionfoundationmodelstemporalpoolinglatentgenerationcoherenceFréchetdistancesemanticspacefutureprediction

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a good video latent space for generation does not have to be learned for pixel reconstruction; it can be inherited, nearly unchanged, from a frozen vision foundation model. V-RAE compresses such frozen features with a lightweight temporal attention pooler and reconstructs video with a spatiotemporal Transformer decoder, and the best variant reaches 2.13 rFVD on K600, the best among the evaluated large-scale pretrained video VAEs, while preserving far more semantic content than VAE latents (89.13% versus 30.83% top-1 UCF101 probing for the strongest baselines). Under matched DiT settings the same latents give gFVD scores of 117.86 on UCF101 and 19.16 on K600, beating all evaluated VAE latent spaces while converging up to 6x faster. The paper further claims that reconstruction fidelity misranks latent spaces and introduces tFVD, a diagnostic that decodes temporally interpolated latents and correlates with generation quality at Pearson 0.919 on K600 versus 0.473 for rFVD. If the claim holds, video tokenizer evaluation should shift from pixel fidelity toward temporal smoothness and semantic organization, and frozen representations become a reusable latent interface for generation and prediction.

What carries the argument

The argument runs on three objects. First, the frozen representation encoder whose feature space defines the latent space itself, so semantic organization is inherited rather than learned; V-RAE works with both frame-wise encoders (DINOv3, SigLIP2, EUPE) and video-native ones (V-JEPA 2.1). Second, the temporal attention pooler, which compresses $r_P$ consecutive encoder features into one latent code using a shared 1D query and zero-initialized temporal biases: initialized to behave like mean pooling, it learns content-adaptive temporal weighting with only about 3M parameters. Third, a MAE-style Transformer decoder with 3D RoPE and multi-frame unpatchify that maps each latent step to $r_{\mathrm{all}}$ consecutive frames, using chunk-causal attention for image encoders and full attention for video encoders. The diagnostic that carries the generative-utility claim is tFVD: replace interior latent codes by temporal midpoints, decode, and measure FVD against real clips, a controlled stress test of whether the latent trajectory stays inside a decodable region under prediction error.

What would settle it

Run the swap experiment on K600: train the DiT on the Wan2.2 VAE latent with the dual-head prediction branch and internal guidance enabled (the V-RAE recipe), and train V-RAE with single-head prediction and internal guidance off, holding latent-token budgets and sampling identical. If the gFVD gap between the two latent spaces collapses under swapped generator recipes, the generator modifications, not the latent organization, drive the reported gains; if a large gap persists, the latent-space attribution survives.

Watch

Extended reading notes

Core claim

The paper's central claim is that a frozen visual representation encoder, a learnable temporal pooler, and a spatiotemporal Transformer decoder jointly define a video latent space that is simultaneously reconstructable, semantically organized, and easy for a diffusion transformer to model. Concretely, V-RAE with V-JEPA 2.1 reports 2.13 rFVD on K600, the best reconstruction among all evaluated large-scale pretrained video VAEs, while retaining near-encoder semantics: SigLIP2-L latents probe at 90.92% top-1 on UCF101 against 30.83% for the strongest VAE baseline, and the best variant achieves 117.86 gFVD on UCF101 and 19.16 on K600 under matched generation settings, with up to 6x faster convergence. The associated diagnostic claim is that rFVD correlates weakly with downstream gFVD (0.200 on UCF101 and 0.473 on K600), whereas the proposed tFVD, FVD computed after replacing interior latent codes with midpoint interpolations $\mathbf{z}'_t=\tfrac{1}{2}(\mathbf{z}_{t-1}+\mathbf{z}_{t+1})$ and decoding, correlates strongly (0.621 and 0.919). The paper reads this as evidence that a generation-friendly latent space must be temporally smooth and decodable under off-trajectory states, not merely faithful at encoded points, and that the same frozen decoder can render both generated and predicted future latents.

Load-bearing premise

The load-bearing premise is the matched-comparison assumption: the reported generation, convergence, and prediction gains are attributed to the latent space itself, which presumes identical DiT recipes across latent spaces, yet V-RAE generators receive a dual-head prediction branch with internal guidance while VAE baselines use single-head prediction without it, and reconstruction compares 16-frame V-RAE clips with 17-frame causal-VAE clips.

Editorial extensions

If this is right

Video tokenizer design can decouple reconstruction from semantics: a frozen vision foundation model plus a small learned pooler and decoder can match or beat large pretrained video VAEs on distributional reconstruction (rFVD) while keeping far more semantic content.
Reconstruction benchmarks mislead: rFVD rankings do not track generation quality, so tokenizer selection should use diagnostics like tFVD that probe the local geometry of the latent space without training a full generator.
A fixed semantic latent interface unifies generation and world modeling: the same frozen decoder renders both DiT-generated latents and predicted future latents, and the Cityscapes results show the predictive variant gains where the baseline drifts.
Optimization cost drops: DiT training in V-RAE latents reaches the gFVD of VAE baselines in roughly one-fifth to one-sixth of the updates, so semantic organization reduces the compute needed to reach a given quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The tFVD protocol is portable: midpoint decodability could be computed for image latents, autoregressive tokenizers, and any future video latent without training a generator, giving the field a cheap pre-screening tool that the paper itself deploys only on its ten compared tokenizers.
The non-affine layer normalization after pooling is a small but general mechanism: by removing the rescaling shortcut in noise-augmented reconstruction, it forces decoders to be robust to injected noise, a trick that could transfer to any noise-augmented autoencoder and may partly explain why V-RAE's decoder tolerates off-trajectory latents.
If the semantic-interface claim holds at scale, video understanding and generation could converge on a single frozen feature space, turning generation into a dynamics problem over semantic states; the paper only tests moderate-scale benchmarks, so open-domain text-to-video remains the open question.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What you should know: this is a serious attempt to transfer the RAE paradigm to video, and the strongest evidence is in reconstruction and semantic probing, not the generation numbers. The generation comparison is not matched: V-RAE's DiT uses dual-head prediction and internal guidance while VAE baselines use single-head prediction with no guidance (Table 3 and Appendix A.3). That asymmetry is load-bearing for the gFVD and 6x-faster-convergence claims. The stress-test note is correct on reading: the paper justifies the asymmetry by citing RAE rather than running a control, so the gains could come from the generator changes instead of the latent space.

What is actually new and good: V-RAE builds a generative latent on frozen VFM features with a lightweight temporal attention pooler and a multi-frame unpatchify decoder. The systematic study across four encoders is well executed. The semantic probing results are the most convincing part — 89% top-1 on UCF101 versus 30% for the strongest VAE baseline is a real gap that survives the temporal compression. The pooling ablation is clean and shows a sensible reconstruction-semantics trade-off. The tFVD diagnostic is clever: replacing interior latents with midpoints and decoding is a nice stress test of local temporal geometry. The authors are also candid in the Limitations section — they acknowledge the moderate scale and open questions about semantic structure.

Soft spots, in order of severity. First, the matched-comparison problem above. The convergence curves (Figure 1, Figure 8) inherit the same asymmetry, so the headline speedup is suspect. Second, Table 1 mixes protocols: V-RAE and AToken use 16 frames, causal VAEs use 17. The 2.13 rFVD on K600 is good but not computed under a single shared protocol. Third, tFVD is validated on the same set of tokenizers used to motivate it (Figure 10), so its superior correlation with gFVD is in-sample. An out-of-sample check — e.g., fitting on UCF101 and testing on K600, or using held-out tokenizers — would strengthen it. Fourth, the Cityscapes future-prediction comparison also uses internal guidance for V-RAE, so the attribution to latent space is not fully clean.

Who this is for: researchers working on video tokenization and latent diffusion. They will find the reconstruction ablations and semantic probing genuinely useful, and the tFVD idea worth building on even if the current validation is incomplete.

My recommendation: send it to peer review. The central direction is plausible and the evidence is extensive. But the authors should be asked to run controls that isolate the latent-space effect — remove the dual-head branch and internal guidance from V-RAE, or add them to the baselines — and to provide an out-of-sample validation of tFVD. Without those, the quantitative claims should be read as upper bounds.

Referee Report

4 major / 5 minor

Summary. The paper proposes V-RAE, a video representation autoencoder in which the latent space is defined by frozen features from a vision foundation model, temporally compressed by a learned attention-pooling module, and decoded by a spatiotemporal Transformer. Reconstruction training combines L1, LPIPS, adversarial, and Gram losses with noise augmentation; generation trains a class-conditional DiT with clean-latent prediction and an auxiliary dual-head objective with internal guidance. The paper evaluates reconstruction (rFVD), semantic probing, class-conditional generation (gFVD), convergence speed, and Cityscapes future prediction against VAE-based tokenizers, and introduces tFVD and TRED as diagnostics. The headline claims are a K600 rFVD of 2.13, gFVD scores of 117.86 on UCF101 and 19.16 on K600, up to 6x faster convergence, and improved future prediction relative to Wan2.2 VAE.

Significance. If the central attribution holds, V-RAE would provide a credible alternative to reconstruction-trained video latents, with the concrete claim that semantic organization and local temporal smoothness make a latent space easier for a downstream generator to learn. The paper is a systematic empirical study rather than a purely proposal-driven one: it includes ablations over temporal-pooling designs, semantic probing across three benchmarks, a matched latent-token budget for the DiT, and a separate future-prediction setup with the same prediction architecture for both latent spaces. These controls, together with an explicit limitations section, are genuine strengths. The main risk is that the headline generation and convergence comparisons are entangled with generator-side enhancements, and that the new tFVD diagnostic is validated on the same set of tokenizers used to motivate it. The direction is plausible and worth publishing after the attribution issues are resolved with additional controls.

major comments (4)

[§2.3, Eq. (12); Table 3; A.3 Table 8] The generation and convergence comparisons are not matched at the generator level. V-RAE's DiT is trained with two prediction heads (Lfull_RF + Lbase_RF in Eq. (12)) and uses internal guidance at inference (1.3 on UCF101 and 1.2 on K600 in Table 8), while the VAE baseline rows in Table 3 are explicitly single-head with no internal guidance, justified by a citation to RAE rather than by a control experiment in this paper. The convergence curves in Figure 1 and Figure 8 inherit the same asymmetry. Without a control that trains VAE baselines with the same dual-head objective and internal guidance, or trains V-RAE without them, the gFVD margins and the '6x faster convergence' claim cannot be attributed to the latent space; they may be generator-side effects.
[Table 1; §3] The headline reconstruction comparison is not computed under a single shared protocol. The table footnote states that V-RAE and AToken use 16-frame inputs while causal video VAEs use 17-frame inputs. Thus the K600 rFVD comparison of 2.13 for V-RAE against 3.58 for Wan2.1 VAE mixes clip-length differences with latent-space differences. The authors should either re-evaluate all tokenizers on the same clip length or explicitly quantify how the one-frame difference affects rFVD; as it stands, the reconstruction superiority claim is not protocol-matched.
[§4.3; Fig. 10; Appendix B] The claim that tFVD is a more reliable predictor of generation quality rests on an in-sample validation. tFVD is introduced with a particular interpolation protocol (Appendix B), motivated by temporal smoothness of V-RAE-style latents, and then correlated with gFVD on the same ten tokenizer configurations that motivated the metric. With only ten points, the Pearson correlations of 0.919 (K600) and 0.621 (UCF101) need confidence intervals, a leave-one-out or out-of-sample analysis, and robustness checks over the protocol choices (e.g., number of interpolated codes, interpolation position, and frame alignment). Without these, the 'tFVD exhibits substantially stronger correlations than rFVD' claim is not yet established beyond this specific set.
[§4.4; Table 4] The Cityscapes future-prediction comparison contains an additional latent-space confound. The paper states that V-RAE 'is also fine-tuned on CoVLA for high-resolution video reconstruction' before being used as the future-prediction latent space, while the Wan2.2 VAE baseline is not fine-tuned on a driving-domain dataset. This gives V-RAE domain-specific adaptation during latent acquisition, so Table 4's gFID/gFVD improvements could reflect the extra fine-tuning rather than the semantic-latent design. The comparison would be clean only if the Wan2.2 VAE received an equivalent amount of domain fine-tuning, or if V-RAE were evaluated without CoVLA fine-tuning.

minor comments (5)

[Abstract] The abstract contains a stray closing brace after '6x faster'; please remove it.
[Table 4] The entry '7.0256319.0233' in the Wan2.2 VAE row appears to be two numbers run together; add a separator or spacing.
[Figure 10] Consider reporting bootstrap confidence intervals for the Pearson correlations, since the number of plotted configurations is small (ten points).
[Figure 1] The axis label 'Compressionvolume ratio' is ambiguous; please clarify whether this is the total compression ratio r_all or a volume-based measure.
[Appendix B] The tFVD protocol compares interpolated reconstructions with ground-truth clips, so tFVD may mix pure reconstruction error with interpolation robustness; please state explicitly whether any decomposition or control is intended.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the core reconstruction, generation, and prediction claims rest on direct empirical comparisons, while the noted protocol asymmetries are validity risks rather than circular reductions.

full rationale

V-RAE's central claims—that frozen semantic features can be decoded into high-quality video reconstructions and can serve as effective DiT latents—are established by direct measurement (Table 1 rFVD, Table 3 gFVD, Table 4 future prediction) rather than derived from the construction of the latent space. The semantic-probing results in Table 2 do track the semantics of the frozen encoders by design, but the paper explicitly reports the drop relative to the frozen encoder and treats the probing outcome as an empirical sanity check, not as a predicted consequence of the method. tFVD is introduced as a fixed diagnostic, not as a fitted proxy for gFVD: its midpoint-interpolation definition (Eq. 14) contains no gFVD term, and no parameter is tuned to maximize the reported Pearson correlations. Computing the correlations on the same tokenizer set used to motivate the metric (Figure 10) is an in-sample validation limitation, but it is not a definitional equivalence between tFVD and gFVD. The generation comparison is not fully matched because V-RAE's DiT uses a dual-head objective and internal guidance (Eq. 12; Appendices A.3 and B) while VAE baselines use single-head prediction without guidance, and the reconstruction protocol evaluates 16-frame versus 17-frame clips. These asymmetries weaken the attribution of the gFVD and convergence gains to the latent space, but they are experimental confounds rather than cases where an input is constructed from the output. No load-bearing claim depends on a self-citation: the RAE/RAEv2 citations are external prior work, and no uniqueness claim is imported from the authors' own prior results. Therefore, no specific circular step can be exhibited under the required standard.

Assumptions & free parameters 6 free parameters · 5 assumptions · 2 invented entities

The central claim depends on two classes of unproven inputs: the invertibility and semantic richness of frozen VFM features, and the validity of I3D-FVD as the evaluation oracle. The free parameters are predominantly hand-chosen protocol and architecture constants; none are fitted to gFVD directly, but several (feature block selection, loss weights, noise range, tFVD protocol) lack sensitivity analyses, so their influence on the reported rankings is unknown.

free parameters (6)

Overall temporal compression ratio r_all = 4 (r_E times r_P; r_P=4 for image encoders, r_P=2 for V-JEPA)
Hand-chosen to fix the latent token budget at 1280 tokens; the same ratio is applied to all variants, so it is a design choice rather than a fitted value, but it determines the information available to the generator.
Selected encoder feature blocks per VFM = DINOv3/SigLIP2: 11,13,15,17,19,21,23; EUPE: 5..11; V-JEPA: 11,13,15,17,19,21,23
The specific transformer blocks are chosen by hand without an ablation study; the chosen blocks affect the semantic content of the latents and hence all downstream metrics.
Reconstruction loss weights = lambda_L1=1, lambda_lpips=1, lambda_gan=0.3, lambda_gram=100
Copied from prior visual tokenizer recipes; these weights shape the decoder and therefore the image-quality and FVD behavior, but no sensitivity analysis is provided.
Latent noise augmentation sigma = U(0, 0.8)
Chosen to make the decoder robust to off-manifold latent codes; no ablation is shown for this range.
tFVD protocol choices = 6 latent codes, midpoint interpolation, comparison to frames 4-19
The number of codes and alignment are fixed without sensitivity analysis; different choices could change the reported correlations.
Internal guidance scales = 1.3 (UCF101), 1.2 (K600, Cityscapes)
Used in V-RAE sampling but not in the single-head VAE baselines; the choice affects gFVD scores and is part of the comparison asymmetry.

assumptions (5)

domain assumption Frozen VFM features contain enough spatial and temporal information for a decoder to reconstruct videos to competitive quality.
The entire reconstruction pipeline depends on the invertibility of the frozen features; the paper tests this empirically on UCF101 and K600 but does not establish it for general video distributions.
domain assumption FVD computed with I3D features is a valid and meaningful measure of video distributional similarity for reconstruction and generation.
rFVD, gFVD, and tFVD all use I3D-based FVD; the paper cites FVD's content bias (Ge et al., 2024) yet still uses FVD as the decisive metric for the headline claims.
domain assumption A DiT with matched backbone and token budget is a fair generator when comparing latent spaces.
This is the central evaluation premise, but Table 3 gives V-RAE dual-head prediction and internal guidance while baselines use single-head prediction, so the premise is only partially satisfied.
domain assumption The Gram, LPIPS, and GAN losses are sufficient to train a decoder that maps latent codes to natural-looking videos.
Standard practice in video tokenizers; no theory is given, and the ability to decode off-manifold latents is critical for generation quality.
domain assumption Temporal attention pooling preserves the semantic organization of the frozen encoder while removing temporal redundancy.
This is the core design claim; it is supported by probing experiments, but the probing protocol itself assumes linear/attentive probes are a valid measure of semantic content.

invented entities (2)

tFVD (Temporal Fréchet Video Distance)
purpose: Diagnostic that decodes interpolated latent midpoints and measures FVD to real clips, intended to forecast downstream generation quality.
Validated only in-sample on the same tokenizers and generator settings used to develop it (Figure 10); no external benchmark, no theoretical derivation, and no sensitivity analysis.
TRED (Temporal Reconstruction Error Difference)
purpose: Measures frame-to-frame fluctuations in reconstruction error over high-frequency regions to visualize reconstruction flicker.
Presented on selected K600 examples only; used for qualitative illustration and not part of the main quantitative claims.

how reviews work

0 comments

Cite this review

Pith. "Pith review of V-RAE: Rethinking Video Latent Spaces for Generation." pith.science (2026). https://pith.science/paper/ZQJQWUFN

@misc{pith2026260813556,
  author       = {Pith},
  title        = {Pith review of: V-RAE: Rethinking Video Latent Spaces for Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZQJQWUFN}},
  note         = {Machine review of arXiv:2608.13556}
}

read the original abstract

Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoder architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization. A reconstruction-optimal latent space, however, need not be well suited to generative modeling. We propose V-RAE, a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from the compressed features. We evaluate V-RAE with four representative frozen encoders on video reconstruction, semantic probing, and class-conditional generation. V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs. Its latents retain substantially more semantic information than conventional video tokenizer latents. Under matched generation settings, our best variant achieves gFVD scores of 117.86 and 19.16 on UCF101 and K600, respectively, while converging up to 6x faster}. We further show that reconstruction quality alone is insufficient to characterize generative utility and introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality. Beyond video generation, V-RAE also improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched prediction settings. Taken together, the experiments show that frozen semantic representations can support video reconstruction, generation, and predictive modeling. The project page: https://v-rae.github.io/.

Figures

Figures reproduced from arXiv: 2608.13556 by the authors.

**Figure 1.** V-RAE overview. Left: Normalized reconstruction, generation, compression, and semantic performance across video tokenizers; FVD axes (↓) are reversed so outward is better. Right: K600 gFVD convergence; V-RAE converges up to 6× faster than VAE-based latent spaces. † We sincerely thank Saining Xie for his direct guidance and valuable feedback, which greatly helped shape V-RAE. arXiv:2608.13556v1 [cs.CV] 13 Aug 2026 [… view at source ↗

**Figure 2.** V-RAE architecture. A frozen visual representation encoder produces temporally dense features, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Temporal Reconstruction Error Difference (TRED) for RAEv2, Wan2.2 VAE, OmniTokenizer, [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (10 more)

**Figure 4.** Figure 4: , each pair uses the same frozen image VFM (DINOv3-L, SigLIP2-L, or EUPE-B), thereby controlling for the underlying semantic representation. RAEv2 reconstructs frames independently, whereas V-RAE couples neighboring features through temporal pooling and decodes them …

**Figure 5.** Figure 5: Reconstruction-semantics trade-off of temporal pooling designs on the UCF101 test set. Point [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 8.** Figure 8: gFVD convergence on UCF101 under matched training settings. V-RAE converges up to 5× faster than Wan2.2 VAE. These convergence results complement the final scores in [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

**Figure 6.** Figure 6: Qualitative class-conditional generation on UCF101. Dashed lines separate the baselines and [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

**Figure 7.** Figure 7: Qualitative class-conditional generation on Kinetics-600. Dashed lines separate the baselines [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

**Figure 9.** Figure 9: Overview of tFVD. (a) Each interior latent is replaced by the midpoint of its two temporal [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

**Figure 10.** Figure 10: Metric-correlation analysis on UCF101 and K600, comparing reconstruction fidelity and [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

**Figure 11.** Figure 11: Qualitative comparison of future video prediction on Cityscapes. We show (a) a traffic scene dominated by vehicle motion and (b) an intersection scene containing pedestrians and bicycles. In each panel, the top row contains six context frames, followed by ground-truth…

**Figure 12.** Figure 12: Structures of the four temporal pooling designs. All modules reduce the encoder features to the [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]

**Figure 13.** Figure 13: Class-conditional generation results for V-RAE with DINOv3-L and SigLIP2-L [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 5 canonical work pages

[1]

Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575,

Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575,

arXiv
[3]

V-JEPA 2: Self-supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985,

Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, et al. V-JEPA 2: Self-supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985,

arXiv
[4]

Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127,

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127,

arXiv
[6]

The latent perturbation is used only as a reconstruction-training augmentation; evaluation, latent-statistics estimation, and latent video generation all use clean latents

LPIPS is evaluated at the native 256× 256 resolution, whereas the Gram branch uses 224× 224 inputs; both perceptual branches cover all 16 frames. The latent perturbation is used only as a reconstruction-training augmentation; evaluation, latent-statistics estimation, and latent video generation all use clean latents. Table 7: Shared VideoMAE discriminator...

work page
[7]

The kinetics human action video dataset.arXiv preprint arXiv:1705.06950,

Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset.arXiv preprint arXiv:1705.06950,

arXiv
[8]

Videopoet: A large language model for zero-shot video generation

Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125,

arXiv
[9]

Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603,

Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603,

arXiv
[10]

Atoken: A unified tokenizer for vision.arXiv preprint arXiv:2509.14476,

Jiasen Lu, Liangchen Song, Mingze Xu, Byeongjoo Ahn, Yanjun Wang, Chen Chen, Afshin Dehghan, and Yinfei Yang. Atoken: A unified tokenizer for vision.arXiv preprint arXiv:2509.14476,

arXiv

Show all 32 references

[11]

Open-MAGVIT2: An open-source project toward democratizing auto-regressive visual generation.arXiv preprint arXiv:2409.04410,

Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang, and Ying Shan. Open-MAGVIT2: An open-source project toward democratizing auto-regressive visual generation.arXiv preprint arXiv:2409.04410,

arXiv
[13]

Vo, Maximilian Seitzer, et al

Oriane Siméoni, Huy V . Vo, Maximilian Seitzer, et al. DINOv3.arXiv preprint arXiv:2508.10104,

arXiv
[14]

Improved baselines with representation autoencoders.arXiv preprint arXiv:2605.18324,

Jaskirat Singh, Boyang Zheng, Zongze Wu, Richard Zhang, Eli Shechtman, and Saining Xie. Improved baselines with representation autoencoders.arXiv preprint arXiv:2605.18324,

arXiv
[16]

UCF101: A dataset of 101 human actions classes from videos in the wild.arXiv preprint arXiv:1212.0402,

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild.arXiv preprint arXiv:1212.0402,

arXiv
[18]

Scaling text-to-image diffusion transformers with representation autoencoders.arXiv preprint arXiv:2601.16208,

Shengbang Tong, Boyang Zheng, Ziteng Wang, Bingda Tang, Nanye Ma, Ellis Brown, Jihan Yang, Rob Fergus, Yann LeCun, and Saining Xie. Scaling text-to-image diffusion transformers with representation autoencoders.arXiv preprint arXiv:2601.16208,
[19]

SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786,

Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and ...

arXiv
[20]

Towards accurate generative models of video: A new metric and challenges.arXiv preprint arXiv:1812.01717,

Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric and challenges.arXiv preprint arXiv:1812.01717,

arXiv
[22]

Larp: Tokenizing videos with a learned autoregressive generative prior

Hanyu Wang, Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava. Larp: Tokenizing videos with a learned autoregressive generative prior. InInternational Conference on Learning Representations, volume 2025, pages 60671–60697,

2025
[23]

Omnitokenizer: A joint image-video tokenizer for visual generation.Advances in Neural Information Processing Systems, 37:28281–28295, 2024a

Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint image-video tokenizer for visual generation.Advances in Neural Information Processing Systems, 37:28281–28295, 2024a. Yuchi Wang, Junliang Guo, Xinyi Xie, Tianyu He, Xu Sun, an...

arXiv
[25]

Making reconstruction fid predictive of diffusion generation fid.arXiv preprint arXiv:2603.05630,

Tongda Xu, Mingwei He, Shady Abu-Hussein, Jose Miguel Hernandez-Lobato, Haotian Zhang, Kai Zhao, Chao Zhou, Ya-Qin Zhang, and Yan Wang. Making reconstruction fid predictive of diffusion generation fid.arXiv preprint arXiv:2603.05630,

arXiv
[26]

Cogvideox: Text-to-video diffusion models with an expert transformer

19 Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. In International Conference on Learning Representations, volume 20...

2025
[27]

Language model beats diffusion–tokenizer is key to visual generation

Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737,

arXiv
[28]

Representation alignment for generation: Training diffusion transformers is easier than you think.arXiv preprint arXiv:2410.06940,

Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think.arXiv preprint arXiv:2410.06940,

arXiv
[29]

Diffusion transformers with representation autoen- coders.arXiv preprint arXiv:2510.11690,

Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoen- coders.arXiv preprint arXiv:2510.11690,

arXiv
[30]

Efficient universal perception encoder.arXiv preprint arXiv:2603.22387,

Chenchen Zhu, Saksham Suri, Cijo Jose, Maxime Oquab, Marc Szafraniec, Wei Wen, Yunyang Xiong, Patrick Labatut, Piotr Bojanowski, Raghuraman Krishnamoorthi, and Vikas Chandra. Efficient universal perception encoder.arXiv preprint arXiv:2603.22387,
[32]

Only the input channel count and corresponding time shift change for EUPE-B

Velocity denominator clamp𝑡𝜖 0.05 0.05 0.05 Full / base loss weights 1.0 / 1.0 1.0 / 1.0 1.0 / 1.0 Online sampling and evaluation Online sampler / steps Euler / 100 Euler / 100 Euler / 100 Online CFG / internal guidance 1.0 / 1.3 1.0 / 1.2 1.0 / 1.2 Evaluation population 2048 ...
[2004]

Representation entanglement for generation: Training diffusion transformers is much easier than you think.Advances in Neural Information Processing Systems, 38:7714–7743, 2025a

Ge Wu, Shen Zhang, Ruijing Shi, Shanghua Gao, Zhenyuan Chen, Lei Wang, Zhaowei Chen, Hongcheng Gao, Yao Tang, Ming-Ming Cheng, et al. Representation entanglement for generation: Training diffusion transformers is much easier than you think.Advances in Neural Information Proces...

2025
[2012]

RoFormer: Enhanced transformer with rotary position embedding.arXiv preprint arXiv:2104.09864,

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding.arXiv preprint arXiv:2104.09864,

arXiv
[2017]

Dera: Decoupled representation alignment for video tokenization.arXiv preprint arXiv:2512.04483,

Pengbo Guo, Junke Wang, Zhen Xing, Chengxu Liu, Daoguo Dong, Xueming Qian, and Zuxuan Wu. Dera: Decoupled representation alignment for video tokenization.arXiv preprint arXiv:2512.04483,
[2018]

Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314,

Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314,

arXiv
[2023]

A short note about Kinetics-600.arXiv preprint arXiv:1808.01340,

Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about Kinetics-600.arXiv preprint arXiv:1808.01340,

arXiv
[2024]

Unitok: A unified tokenizer for visual generation and understanding.Advances in Neural Information Processing Systems, 38: 129274–129297, 2025a

Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiaojuan Qi. Unitok: A unified tokenizer for visual generation and understanding.Advances in Neural Information Processing Systems, 38: 129274–129297, 2025a. Xin Ma, Yaohui Wang, Gengyun Jia, ...

arXiv
[2025]

CoVLA: Comprehensive vision-language-action dataset for autonomous driving

Hidehisa Arai, Keita Miwa, Kento Sasaki, Kohei Watanabe, Yu Yamaguchi, Shunsuke Aoki, and Issei Yamamoto. CoVLA: Comprehensive vision-language-action dataset for autonomous driving. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1933–1943,

1933
[2026]

Improving the diffusability of autoencoders.arXiv preprint arXiv:2502.14831,

Ivan Skorokhodov, Sharath Girish, Benran Hu, Willi Menapace, Yanyu Li, Rameen Abdal, Sergey Tulyakov, and Aliaksandr Siarohin. Improving the diffusability of autoencoders.arXiv preprint arXiv:2502.14831,

arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575,

Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575,

arXiv

[2] [3]

V-JEPA 2: Self-supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985,

Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, et al. V-JEPA 2: Self-supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985,

arXiv

[3] [4]

Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127,

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127,

arXiv

[4] [6]

The latent perturbation is used only as a reconstruction-training augmentation; evaluation, latent-statistics estimation, and latent video generation all use clean latents

LPIPS is evaluated at the native 256× 256 resolution, whereas the Gram branch uses 224× 224 inputs; both perceptual branches cover all 16 frames. The latent perturbation is used only as a reconstruction-training augmentation; evaluation, latent-statistics estimation, and latent video generation all use clean latents. Table 7: Shared VideoMAE discriminator...

work page

[5] [7]

The kinetics human action video dataset.arXiv preprint arXiv:1705.06950,

Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset.arXiv preprint arXiv:1705.06950,

arXiv

[6] [8]

Videopoet: A large language model for zero-shot video generation

Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125,

arXiv

[7] [9]

Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603,

Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603,

arXiv

[8] [10]

Atoken: A unified tokenizer for vision.arXiv preprint arXiv:2509.14476,

Jiasen Lu, Liangchen Song, Mingze Xu, Byeongjoo Ahn, Yanjun Wang, Chen Chen, Afshin Dehghan, and Yinfei Yang. Atoken: A unified tokenizer for vision.arXiv preprint arXiv:2509.14476,

arXiv