{"id":"26fdb1ab-b8ba-4ff8-a6ef-e8d771fa536b","arxiv_id":"2607.14088","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Frozen video foundation features can be compressed into reconstruction-capable, generation-friendly latents that improve video generation quality and convergence speed.","lead":"VideoRAE replaces the usual train-from-scratch video autoencoder with a frozen video foundation model as encoder, then compresses its features into latents that work for both diffusion and autoregressive generation. It reports state-of-the-art UCF-101 generation scores and roughly five times faster convergence of downstream generators.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"REPA's claimed latent-manifold regularization is asserted, not measured; the central KL-free mechanism is supported only by downstream metrics that do not isolate latent geometry.","rationale":"The reader's weakest assumption identifies exactly the spot where the paper's central mechanistic claim is least supported: Sec. 3.3 equates the REPA objective with latent-geometry regularization without any direct geometric evidence. The ablation in Table 7 controls the decoder/GAN/generator pipeline, so it shows REPA is doing something, but it does not show that what it does is to make the latent space prior-friendly. That distinction matters because the paper explicitly argues that semantic topology, not pixel fidelity, drives generation quality, and because the headline convergence gains are attributed to this topology. I therefore agree with the conditional verdict. I would not move the verdict to reject: the empirical gFVD/reconstruction results, especially the REPA on/off ablation, provide reasonable indirect support and the work is reproducible in principle. The requested latent-space measurement is a modest, feasible addition that would either validate the mechanism or force a more cautious claim. No internal inconsistency or fraud is suggested; the paper simply states a mechanism without measuring it. My recommended action is to keep CONDITIONAL and ask for the latent-geometry analysis plus code/error bars before accepting the KL-free and 5x-convergence claims at face value.","tokens_in":15282,"tokens_out":8714,"duration_ms":89069,"concrete_test":"For the continuous VideoRAE(V-JEPA2) model and its 'w/o REPA' ablation, extract z_cont on 10k held-out clips; estimate the per-dimension marginal kurtosis, covariance eigenspectrum, and the 99th-percentile Mahalanobis distance of training latents. Then sample 10k latents from the trained DiT and measure the same statistics, along with the fraction of generated latents outside the training typical set. If REPA-on is not closer to a smooth, full-rank, isotropic prior than REPA-off, and DiT samples are equally out-of-distribution, the KL-free 'well-behaved manifold' claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that frozen VFM features yield generation-friendly latents rests on Sec. 3.3's assertion that the REPA loss (Eq. 6) 'implicitly regularizes the geometry of the latent space' and 'completely replaces the traditional KL-divergence constraint.' No measurement of the latent distribution, smoothness, or topological regularity is reported. Table 7 shows that adding L_REPA improves gFVD (105→93 continuous, 67→40 discrete), but this is a downstream metric; it cannot distinguish whether REPA makes the latent manifold prior-friendly or simply acts as a feature-space reconstruction/consistency loss that improves decoder features. The convergence comparisons in Fig. 7 inherit this ambiguity: if the latent prior is not actually well-behaved, the faster DiT/AR convergence could be due to the autoencoder's decoder/GAN losses or the specific generator capacity rather than the semantic topology of the frozen VFM latents. The paper never reports error bars or code, and no limitations section addresses this missing measurement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VideoRAE, a video autoencoder that replaces the conventional learnable 3D encoder with a frozen Video Foundation Model (VFM) whose multi-scale hierarchical features are compressed by a lightweight 1D self-attention projector. The resulting latents support both continuous (for DiTs) and discrete (via a Multi-Codebook SimVQ) generative models. A Representation Alignment (REPA) loss aligns decoder features with the frozen VFM teacher and is claimed to replace the KL divergence of standard VAEs. The paper reports strong reconstruction and state-of-the-art class-to-video gFVDs on UCF-101 (40 discrete AR, 93 continuous DiT), about 5× faster convergence than LARP/LTX-VAE baselines, and a controlled 2B-scale text-to-video study showing better VBench scores than LTX-VAE.","tokens_in":15624,"tokens_out":3718,"duration_ms":40146,"significance":"If the results hold, VideoRAE would be a significant step toward unifying video understanding and generation by demonstrating that frozen VFM features can be directly reused as compact, reconstruction-capable, and generation-friendly latent spaces. The paper is well positioned in a timely research direction and includes extensive ablations over quantization, REPA, feature layers, and fusion strategies, which is a strength. However, the central KL-free claim rests on an unmeasured assertion about latent geometry, and the empirical comparisons rely on single-run metrics without code or checkpoints. These limitations currently temper the strength of the contribution, but the core idea is plausible and worthy of further validation.","major_comments":[{"comment":"The paper's core mechanistic claim is that REPA 'implicitly regularizes the geometry of the latent space' and 'completely replaces the traditional KL-divergence constraint.' This is asserted, not demonstrated. The only evidence is downstream gFVD improvements in Table 7, but gFVD is a generation metric that could improve because of better decoder features, GAN losses, or generator capacity, without implying that the latent manifold is actually prior-friendly or topologically regular. Since the KL-free mechanism is a central contribution, the authors should provide direct evidence: e.g., measure the empirical latent distribution against the assumed Gaussian, analyze latent nearest-neighbor/interpolation behavior, or compare against a KL-regularized variant with the same decoder and GAN losses. The current omission is load-bearing for the claim that VideoRAE's convergence speed and SOTA ge","section":"§3.3, Eq. (6)"},{"comment":"The controlled T2V comparison is confounded by a frame-count mismatch: LTX-VAE is trained on 17-frame clips and VideoRAE on 16-frame clips. This changes the token sequence length, temporal receptive field, and effective compute per sample. The paper states the settings are 'comparable,' but the difference makes the comparison not controlled. To support the faster-convergence claim, both models should be trained on the same clip length, or the paper should explicitly show that the one-frame difference is immaterial (e.g., by ablating LTX-VAE on 16 frames). Without this, the VBench advantage and convergence gap cannot be attributed cleanly to the latent space.","section":"§4.1, 'Text-to-Video Generation on VBench' and Table 5"},{"comment":"All quantitative claims—especially 'state-of-the-art gFVD' and 'approximately 5× faster convergence'—are based on single runs without variance or significance testing. The differences between some configurations are small (e.g., Table 4: 93 vs 94 for 1024×64 between V-JEPA and VideoMAEv2), and the convergence comparison in Fig. 7 uses different model sizes and token budgets (VideoRAE AR is 1.3B vs LARP 632M). No code or checkpoints are available to reproduce the numbers or verify the 'identical protocols' claim for baselines. This is a significant reproducibility concern that undermines the strength of the central quantitative claims.","section":"Tables 3–4, Fig. 7"},{"comment":"There is no limitations section, and the paper does not acknowledge the absence of direct latent-geometry measurements or the confounds in the convergence comparison. The 'Discussion' paragraph in §4.1 addresses reconstruction vs. generation quality but does not flag these methodological limitations. Given the central claims, the authors should add a dedicated limitations discussion or, better, supply the missing experiments to address the concerns above.","section":"§5 (Conclusion) and overall manuscript"}],"minor_comments":[{"comment":"The manuscript contains several typos and formatting issues: 'W AN2.1-V AE' should be 'WAN2.1-VAE', 'V AEs' should be 'VAEs', '16×256×256' appears inconsistently, and the title of Table 4 is missing a closing parenthesis. These are cosmetic but should be cleaned.","section":"Throughout"},{"comment":"The 1D Projector is described only as 'based on self-attention mechanisms.' Please specify the number of layers, heads, and dimension, as this module is a key component for the compression and its design choices affect the latent capacity.","section":"§3.1, Eq. (2)"},{"comment":"Hyperparameters such as the REPA loss weight λ (Eq. 6) and λ_repa (Eq. 7) are not reported. The paper also does not state which V-JEPA 2 / VideoMAEv2 model version is used. Please provide these details for reproducibility.","section":"§4, 'Implementation details'"},{"comment":"Several references carry future-dated arXiv IDs (e.g., '2603.14482', '2601.03233', '2506.09985' with 2025 date), which is unusual and should be checked for correctness.","section":"References"},{"comment":"The evaluation protocols use different numbers of generated samples (10K for discrete AR, 2K for continuous DiT). This makes cross-paradigm gFVD comparisons difficult and should be noted explicitly.","section":"Tables 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important problem, and the proposed VideoRAE architecture is plausible. However, the central 'KL-free latent manifold' claim is not directly supported by measurements, and the T2V comparison is confounded by a frame mismatch. The lack of code and single-run metrics also prevents verification of the SOTA claims. These issues are addressable in revision, but they require substantial additional evidence rather than simple presentation fixes. I would recommend major revision and specifically ask for direct latent-geometry analysis, a de-confounded T2V comparison, and reproducibility details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is worth taking seriously: use a frozen video foundation model as the encoder of a video autoencoder, compress its features with a 1D projector, and replace the usual KL prior with a representation-alignment loss. As far as I know this is the first systematic video instantiation of the representation-autoencoder idea, and the paper does it cleanly. The multi-codebook SimVQ strategy is a sensible way to keep high-dimensional semantic features quantizable, and the local-global REPA loss is a reasonable way to align decoder features with the teacher. The ablations are genuinely useful—quantization method, feature aggregation, fusion type—and the observed tradeoff between V-JEPA 2 (better generation) and VideoMAEv2 (better reconstruction) is a nice empirical finding that supports the paper's main thesis. Where it gets soft: the paper claims in Sec. 3.3 that REPA implicitly regularizes the geometry of the latent space and completely replaces the KL divergence. That is asserted, not demonstrated. Table 7 shows REPA improves gFVD substantially (105 to 93 continuous, 67 to 40 discrete), but gFVD is a downstream metric. It cannot distinguish the latent manifold is prior-friendly from the decoder features are better or the GAN loss is doing more of the work. No measurement of latent space structure appears anywhere. This is the load-bearing gap, and it needs to be addressed directly—e.g., covariance spectrum, nearest-neighbor structure, or a prior-matching diagnostic. Other issues are less severe but real. The T2V comparison uses 17-frame clips for LTX-VAE and 16-frame for VideoRAE; the paper says this matches each autoencoder's native interface, but it is still a confound that weakens the controlled claim. There are no error bars anywhere, and no code or checkpoints at submission, which matters for a system paper claiming 5x faster convergence on the basis of a single training run. Also, the reconstruction tables show industry VAEs like WAN2.1 beat VideoRAE on PSNR/rFVD; that is fine if the case is about generation-friendliness, but it does mean the central advantage needs to be demonstrated robustly. On the circularity front: the REPA teacher being the same frozen VFM as the encoder is a design choice, not a circular fit. The compressed latent must still reconstruct pixels, so the result is meaningful. Who should read this: people working on video tokenization, latent-space design, and foundation-model representations. It deserves a serious referee, not a desk reject. But I would send it back with requests for code, error bars, a matched-frame T2V rerun, and some direct look at the latent manifold. The direction is solid; the proof is not complete.","headline":"A plausible, useful video-representation autoencoder whose central KL-free claim rests on an unmeasured latent-manifold assumption; worth refereeing, but the evidence needs to be much tighter.","tokens_in":806,"tokens_out":895,"would_cite":true,"duration_ms":35723,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Frozen video foundation models, built for understanding, can be repurposed as the encoder of a video autoencoder, producing latent spaces that yield state-of-the-art class-to-video generation and roughly five-fold faster generative training","keywords":["video representation autoencoder","frozen video foundation models","latent video generation","representation alignment","multi-codebook quantization","autoregressive video generation","diffusion transformers","video latent compression"],"falsifier":"Train the same VideoRAE with a conventional KL loss added in place of the REPA loss and compare gFVD and convergence; if the scores stay near 93/40, then REPA's semantic regularization is not the cause. Alternatively, measure the latent-space geometry directly—for example, interpolate between two V-JEPA 2 latents and reconstruct; if interpolations collapse to unrealistic videos, the latent manifold is not as smooth as claimed.","tokens_in":15232,"feed_emoji":"🎬","tokens_out":5522,"duration_ms":47650,"temperature":0.7,"pith_summary":"This paper asks whether the semantic representations from a frozen video foundation model—one trained for understanding, not reconstruction—can be turned into a compact latent space for video generation. The authors answer yes with VideoRAE, which uses the frozen model as a fixed encoder, compresses its multi-scale features with a 1D self-attention projector, and supports both continuous latents for diffusion transformers and discrete multi-codebook tokens for autoregressive models. A representation-alignment loss replaces the usual KL regularization, and the paper reports state-of-the-art gFVDs of 40 (autoregressive) and 93 (diffusion) on UCF-101, with generative training converging about five times faster than with conventional pixel-driven autoencoders. If correct, this means video generation can inherit semantic structure from understanding models instead of learning it from scratch.","feed_headline":"Frozen video models can drive video generation, 5x faster training","feed_subtitle":"VideoRAE reuses frozen video features as latents, beating pixel-driven autoencoders on UCF-101.","key_machinery":"The central machinery is the frozen video foundation model used as a fixed semantic encoder, paired with a 1D self-attention projector that compresses its multi-scale features into a compact token sequence. The REPA (representation alignment) loss aligns decoder features with the frozen teacher at local and global scales, which the paper argues regularizes the latent manifold and eliminates the need for KL regularization. For the discrete branch, Multi-Codebook SimVQ splits the latent into K sub-vectors, each quantized against a learnable MLP-mapped frozen base codebook, preserving high dimensionality and avoiding codebook collapse.","core_discovery":"The paper claims that a frozen, self-supervised video foundation model—trained for understanding, not generation—can serve as the encoder of a video autoencoder. VideoRAE fuses hierarchical features from that frozen model, compresses them with a lightweight 1D self-attention projector into 512 or 1024 tokens, and maps them to either a continuous latent (for diffusion transformers) or discrete multi-codebook tokens (for autoregressive models). A representation-alignment loss, computed by matching decoder features to the frozen teacher's features locally and globally, replaces the KL divergence used in conventional VAEs. On UCF-101 class-to-video generation, the resulting latents give gFVD 40","pith_inferences":["If this generalizes, future video autoencoders could be upgraded by swapping in a better frozen foundation model without redesigning reconstruction, letting representation-learning progress feed directly into generation quality.","The paper's observation that a lower-reconstruction latent produces better generation suggests a re-weighting of autoencoder objectives: semantic and structural alignment may be worth trading against PSNR.","A testable extension: apply the same frozen-encoder-plus-REPA recipe to other modalities or tasks, such as audio or 3D, where self-supervised models exist but generation still relies on pixel-level autoencoders.","The absence of a KL prior means the generator must match the latent distribution induced by the VFM; it would be worth measuring how close that distribution is to a simple prior, and whether the 5x speedup persists when the VFM is changed."],"forward_implications":["If the central claim holds, video generation no longer requires training a pixel-reconstruction encoder from scratch: a frozen understanding model can provide the latent space, simplifying the autoencoder and lowering compute.","Generative models trained on these latents converge roughly 5x faster than on conventional video tokenizers, which cuts the training cost of video diffusion and autoregressive models.","Because the same frozen encoder supports both continuous and discrete latents, one autoencoder can serve both diffusion-transformer and autoregressive pipelines, unifying the two paradigms.","For generation quality, semantic fidelity matters more than pixel fidelity: the model with slightly lower reconstruction (V-JEPA 2-based) yields better gFVD than the one with higher PSNR (VideoMAEv2-based).","The approach extends to text-to-video at scale: replacing an existing video VAE with VideoRAE in a 2B-scale T2V system improves VBench scores and convergence."],"fun_headline_variants":["Frozen video model now drives video generation","VideoRAE: reuse frozen video model as autoencoder","Frozen video features replace pixel-based VAEs","One frozen encoder for AR and diffusion video models","Frozen video model powers 5x faster video gen"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole speedup and quality gain rests on the claim that the representation-alignment loss alone, without any KL term, keeps the latent space smooth and prior-friendly; the paper offers no direct measurement of that geometry, only downstream metrics, so if alignment does not actually impose that structure, the gains could be caused by the stronger decoder or adversarial loss instead.","fun_headline_variants_meta":{"raw":{"variants":["Frozen video model now drives video generation","VideoRAE: reuse frozen video model as autoencoder","Frozen video features replace pixel-based VAEs","One frozen encoder for AR and diffusion video models","Frozen video model powers 5x faster video gen"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001034,"raw_usage":{"total_tokens":4244,"prompt_tokens":848,"completion_tokens":3396,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":3322}},"tokens_in":592,"tokens_out":3396,"duration_ms":24135,"temperature":1.0,"reasoning_tokens":3322,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T02:46:41.816273+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same VideoRAE with a conventional KL loss added in place of the REPA loss and compare gFVD and convergence; if the scores stay near 93/40, then REPA's semantic regularization is not the cause. Alternatively, measure the latent-space geometry directly—for example, interpolate between two V-JEPA 2 latents and reconstruct; if interpolations collapse to unrealistic videos, the latent manifold is not as smooth as claimed.","supporting_citations":[],"review_version":1}