{"id":"73a6cce5-976d-4ef5-95b8-de697c0b3e81","arxiv_id":"2412.04452","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A four-plane factorized video latent space cuts the diffusion model sequence length from 1280 to 672 tokens, achieving FVD 38 versus 39 for the volumetric W.A.L.T. baseline on UCF-101 at roughly half the compute.","lead":"This paper introduces a video autoencoder that compresses video into four flat planes instead of a full 3D block, nearly halving the latent size. Paired with a diffusion model, this makes video generation about twice as fast while staying close to the volumetric baseline in quality.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The head-to-head generation quality claim hinges on a reimplemented W.A.L.T.* baseline whose FVD deviates by 7 points from the original (39 vs 46), unexplained in the paper.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the faithful reproduction of the W.A.L.T. baseline. The paper's own numbers show an unexplained 7-point FVD discrepancy between W.A.L.T.* and the original W.A.L.T., which directly undermines the only head-to-head volumetric baseline used to support the central 'comparable quality' claim. Other issues (no error bars, test-set hyperparameter selection, undisclosed internal data for interpolation) are real but secondary; they affect confidence intervals and generalization but do not single out the core comparison as sharply as the baseline discrepancy. The speedup claim is robust because it is measured directly (380 ms vs 750 ms per step) and via FLOPs, independent of baseline quality. However, without a verified baseline, the paper cannot substantiate that the factorized representation matches volumetric generation quality, which is the second half of the central claim. The recommended verdict remains CONDITIONAL: the paper should release code/checkpoints and explain or correct the W.A.L.T.* discrepancy. This does not change the reader's verdict, but it reinforces the condition.","tokens_in":15424,"tokens_out":8323,"duration_ms":155135,"concrete_test":"Obtain the official W.A.L.T. code or pretrained checkpoints (from the original authors if available) and evaluate the unmodified model on UCF-101 at 128x128 with the exact FVD protocol and sample count used for 4Plane. Compare this score to the paper's W.A.L.T.* (39) and original W.A.L.T. (46). If the official evaluation yields ~39, the reimplementation is faithful and the head-to-head stands; if it yields ~46, W.A.L.T.* is not representative, and all 'comparable or better' claims must be re-run against a correctly reproduced baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that four-plane factorized latents achieve generation quality comparable to volumetric W.A.L.T. while being ~2x faster. All head-to-head quality comparisons use W.A.L.T.*, the authors' reimplementation, which reports FVD 39 on UCF-101 128x128 versus the original W.A.L.T.'s 46 (Table 2). Section 4.1 states 'we were able to reproduce the model in terms of similar datasets and performance,' but the 7-point gap contradicts this assertion. If W.A.L.T.* is not a faithful reproduction—due to missing architectural details, different training data, or a different evaluation protocol—then the 'comparable' quality claim is not established against the actual W.A.L.T. method. The paper neither explains the discrepancy nor releases code or checkpoints, so the fidelity of the baseline cannot be verified independently. Since the efficiency gain (sequence length 672 vs 1280, ~2x step time) is not in dispute, the entire 'without sacrificing quality' component of the central claim rests on this single, unvalidated baseline comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a four-plane factorized video latent representation, replacing the traditional volumetric latent volume with two spatial planes and two spatiotemporal planes. For a 128x128, 17-frame video, the resulting transformer sequence length drops from 1,280 to 672 (Table 1). The authors evaluate reconstruction quality on Kinetics-600, class-conditional generation on UCF-101, frame prediction on K600, and two-frame interpolation on DAVIS-7/UCF-7, reporting generation FVD values comparable to a reimplemented W.A.L.T. baseline (W.A.L.T.*) while achieving roughly 2x faster diffusion training (380 ms vs 750 ms per iteration). The paper includes ablations of the factorization and combine operations, a tri-plane comparison, and timing measurements across several hardware configurations.","tokens_in":15590,"tokens_out":5296,"duration_ms":49879,"significance":"If the empirical claims hold, this is a useful efficiency contribution: the four-plane factorization preserves reconstruction fidelity and competitive generation quality while substantially shortening the sequence length presented to a transformer-based diffusion model. The paper is a purely empirical study with careful attention to ablations (Sec. 4.5, Appendix D) and hardware timing (Appendix C), which are strengths. However, the central 'without sacrificing quality' claim is not yet securely established. The head-to-head generation comparison rests on a reimplemented W.A.L.T.* baseline whose FVD differs from the published W.A.L.T. by 7 points (39 vs 46, Table 2), with no explanation. In addition, the key UCF-128 comparison is a one-point FVD gap with no error bars, and the frame-prediction result shows a clear degradation against the published W.A.L.T. (8.6 vs 3.3). These issues make the central claim quantitatively fragile despite the clear efficiency benefit.","major_comments":[{"comment":"The paper states in Sec. 4.1 that 'we were able to reproduce the model in terms of similar datasets and performance,' but Table 2 reports W.A.L.T.* with FVD 39 on UCF-101 128x128, whereas the original W.A.L.T. paper reports 46. This 7-point gap is unexplained. The central 'comparable to W.A.L.T.' claim is based on the 38 vs 39 comparison against this reimplementation, not against the published system. The authors need to explain the discrepancy (architecture changes, training data, evaluation protocol) or compare against the official W.A.L.T. checkpoint; otherwise the headline quality comparison is not established.","section":"Sec. 4.1 / Table 2"},{"comment":"The UCF-128 class-conditional FVD comparison is 38 (4Plane) vs 39 (W.A.L.T.*), a one-point difference. FVD is a distributional metric with sample noise, and no error bars, confidence intervals, or multiple-seed results are reported. The text concludes the results are 'comparable,' but a single one-point difference without variance characterization is not sufficient support. Please report variance across seeds or another statistical quantification.","section":"Sec. 4.2.2 / Table 2"},{"comment":"For frame prediction on Kinetics-600, the reported FVD is 8.6 for 4Plane, 5.7 for W.A.L.T.*, and 3.3 for the original W.A.L.T. This is a material degradation, yet Sec. 4.3 states the model is 'comparable to W ALT*.' The gap is roughly 50% relative to the original W.A.L.T. and should be discussed honestly, including whether the efficiency gain justifies this quality loss.","section":"Sec. 4.3 / Table 2"},{"comment":"The 256x256 result (4Plane FVD 58.27 vs W.A.L.T.* 84.68) is presented as evidence that a shorter sequence 'can in fact reduce the modeling burden on the denoiser network.' This is a strong interpretative claim, but no analysis is provided to support it (e.g., training curves, controlled sequence-length ablations, or diagnostics). Given that the reimplementation itself is suspect at 128x128, this result does not yet provide a robust basis for that conclusion. Please provide additional evidence or temper the claim.","section":"Sec. 4.2.2 / Table 2"}],"minor_comments":[{"comment":"The text contains a typo: 'apendix' should be 'appendix.'","section":"Sec. 4.1"},{"comment":"The table note contains typos: 'exeperiments' and 'resolutios' should be 'experiments' and 'resolutions.' The notation 'W ALT*' is also awkward; consider 'W.A.L.T.*' for clarity.","section":"Table 2"},{"comment":"The section intro states the factorized autoencoder design is used 'without modification across experiments,' but Sec. 3.1.1 defines different constructions for the spatial planes: Eq. (3) uses time-split aggregation for class-conditional generation and frame prediction, while Eq. (4) uses encoded boundary frames for interpolation. Please clarify whether the design changes or only the input source to the planes changes.","section":"Sec. 4 (intro)"},{"comment":"The joint image-video training strategy in Appendix F is explicitly untested ('While we have not experimented with it'). This is fine as speculation, but it should be labeled as a proposed extension rather than a claim about the method's demonstrated capabilities.","section":"Appendix F"},{"comment":"The timing statement '380 ms compared to 750 milliseconds for W.A.L.T.' is ambiguous about batch size. Appendix C says the reported timings correspond to a batch size of 256 but then says they approximately align with the batch-size-8 row of Figure 5. Please make the reported hardware and batch-size conditions internally consistent.","section":"Sec. 4.2.2 / Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the W.A.L.T.* baseline fidelity. If the authors can obtain the original W.A.L.T. checkpoint or provide a detailed, convincing explanation for the 7-point FVD discrepancy, the paper's efficiency contribution is likely sufficient. However, the current draft overstates the 'without sacrificing quality' claim, and the lack of error bars on the key comparisons weakens the quantitative support. I would recommend a major revision with a request to either strengthen the baseline comparison or revise the claims to match the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The four-plane factorized autoencoder is a useful efficiency tool. It cuts the latent sequence length from 1280 to 672 for 128x128, 17-frame video, and the measured ~2x training speedup is credible. On their own reimplementation of W.A.L.T. (W.A.L.T.*), the generation FVD is essentially tied (38 vs 39 on UCF-101). The caveat: W.A.L.T.* differs from the original W.A.L.T. by 7 FVD points (46 vs 39) and the paper never explains why.\n\nWhat is new: the four-plane factorization extends tri-plane video latents by adding a second spatial plane, which also enables frame prediction and interpolation. The task-adaptive definition of the spatial planes (split along time for class-conditional/frame prediction, boundary frames for interpolation) is a sensible design. The paper does solid empirical work: multiple tasks, ablations (mean pooling vs linear projection, concat vs sum, tri-plane vs four-plane), and a timing study across TPU/GPU. Reconstruction quality is nearly unchanged despite the compression.\n\nSoft spots, in proportion: the W.A.L.T.* issue is the biggest. The paper says they reproduced the model with similar performance, but the 7-point gap contradicts that. If the reimplementation is actually better than the original, then the \"comparable to W.A.L.T.\" claim is really \"comparable to our improved W.A.L.T.\" and the original number is not the right reference. That said, the 4Plane method also beats older published baselines (MAGVIT-v2, PVDM), so quality is not anchored solely on W.A.L.T.*. Still, the discrepancy needs an explanation. Minor issues: no error bars or significance for the headline FVD differences, and design choices (number of planes, combine op, pooling) are selected on UCF-101, the same benchmark used for final evaluation. The interpolation experiments use an internal dataset with no release, so that part is hard to verify.\n\nWho this is for: anyone working on latent video diffusion who cares about compute. The speedup is real and the quality tradeoff looks modest. The paper deserves a serious referee. I would ask for code/checkpoints, variance estimates, and a clear explanation of W.A.L.T.* before accepting it unconditionally.","headline":"Four-plane factorization delivers a real ~2x speedup for latent video diffusion, but the unexplained 7-point FVD gap in the reimplemented W.A.L.T. baseline keeps the quality-parity claim from being fully established.","tokens_in":16177,"tokens_out":3092,"would_cite":true,"duration_ms":30167,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Four-plane video latents cut diffusion sequence length from t·h·w to t·(h+w)+2·h·w, and the paper shows this compressed latent still yields generation FVD on par with the volumetric W.A.L.T. baseline while training roughly twice as fast.","keywords":["video generation","latent diffusion","factorized representation","video autoencoder","tri-plane","transformer diffusion","video tokenization","efficient generative modeling"],"falsifier":"Train the same transformer diffusion model on the volumetric latent with the same data and compute, and measure FVD on UCF-101 128×128: if the volumetric model matches or beats the factorized model's 38—or if the original W.A.L.T.'s published 46 is reproduced instead of the 39 reported for W.A.L.T.*—then the claim that four-plane factorization preserves generation quality is undermined. The paper's own Table 4 offers a second check: swapping mean pooling for linear projection worsens FVD to 50, showing the result depends on that design choice.","tokens_in":15168,"feed_emoji":"🎬","tokens_out":8822,"duration_ms":72806,"temperature":0.7,"pith_summary":"This paper tries to establish that a video autoencoder can compress the volumetric latent t×h×w into four axis-aligned planes—two spatial and two spatiotemporal—whose combined size t×(h+w)+2×h×w grows sublinearly with the input volume, and that this compact latent still carries enough information for a transformer-based latent diffusion model to generate videos of quality comparable to the volumetric baseline. The claim matters because video diffusion is bottlenecked by sequence length: if the factorization preserves fidelity, it offers a drop-in efficiency upgrade for class-conditional generation, frame prediction, and video interpolation. The paper reports reconstruction PSNR of 27.11 vs 27.64 for the volumetric W.A.L.T. baseline at 128×128 with 17 frames, and generation FVD of 38 vs 39 on UCF-101 for the re-implemented W.A.L.T.* baseline, while cutting training iteration time from 750 ms to 380 ms.","feed_headline":"Four-plane video latents halve diffusion cost","feed_subtitle":"A 672-token latent replaces a 1280-token volume, matching generation FVD while training 2x faster.","key_machinery":"The machinery is the four-plane factorization of the latent volume Z∈$R^{{t×h×w×c}}$: two spatial planes $P^{1}$_xy, $P^{2}$_xy∈$R^{{h×w×c}}$ come from splitting the time axis and mean-pooling each half, while two spatiotemporal planes P_xt∈$R^{{t×h×c}}$ and P_yt∈$R^{{t×w×c}}$ come from mean-pooling along width and height. The decoder recomposes the volume by querying the four planes at each (x,y,t) and concatenating the features, then feeds the rebuilt volume through a 3D CNN decoder. This factorization is what turns the transformer sequence of length t×h×w into one of length t×(h+w)+2×h×w, and the paper's ablations show concatenation over summation and mean pooling over learned linear projection as the best choices.","core_discovery":"The paper's central claim is that volumetric video latents can be factorized into four planes—two temporal-mean spatial planes and two spatial-mean temporal planes—and rebuilt by concatenating the four queried plane features at each voxel, losing only a small amount of reconstruction fidelity while almost halving the diffusion sequence length (1280 to 672 tokens at 128×128). The paper shows this factorized space supports a transformer-based latent diffusion model whose generation FVD is on par with the volumetric baseline on UCF-101 class-conditional generation (38 vs 39 for W.A.L.T.* at 128×128, and 58.27 vs 84.68 at 256×256 in the factorized model's favor), while on K600 frame prediction it trails (8.6 vs 5.7). The four-plane design is framed as a fix for tri-plane latents' information mixing: two spatial planes retain more spatial structure and make frame-conditional tasks straightforward.","pith_inferences":["The sublinear scaling suggests the factorization becomes more attractive for longer and higher-resolution videos; the 56-frame experiment hints at this, but the paper does not test 128-frame or higher-resolution regimes.","Appendix F sketches a joint image-video training strategy that discards one redundant spatial plane for images; if that works, the same tokenizer could unify image and video diffusion training with only a small sequence-length increase (288 vs 256 tokens for a 16×16 latent).","Because the spatial planes are time-averaged, the representation may be lossy for fast, high-frequency motion; the comparable FVD on UCF-101, whose actions are mostly slow, does not guarantee the same on datasets with rapid motion, which would be a natural stress test.","The factorization and recombination are simple pooling and concatenation operations, so the representation could be ported to other 3D volumetric data domains (e.g., medical volumes or 3D scenes) wherever a transformer operates on a flattened volume."],"forward_implications":["At 128×128, the diffusion sequence length drops from 1280 to 672, giving roughly 2× faster training iterations (380 ms vs 750 ms) and 2.4× faster inference per video (0.17 s vs 0.40 s).","At 256×256, the sequence length stays at 672 while the volumetric baseline grows to 1280, and generation FVD improves over W.A.L.T.* (58.27 vs 84.68 on UCF-101).","The same factorized autoencoder supports class-conditional generation, frame prediction (using one spatial plane as conditioning), and video interpolation (conditioning on both boundary-frame spatial planes), with interpolation FVD 156.1 on DAVIS-7 vs 199.3 for VIDIM.","Longer videos become feasible: 56-frame generation runs at 378 ms per step while the volumetric baseline exceeds memory limits.","The factorization holds in both AE and VAE settings (with W.A.L.T.-style and WF-VAE-style autoencoders), so the efficiency gain is not tied to a particular autoencoder loss."],"supporting_citations":[{"why":"The W.A.L.T. model is the volumetric baseline for reconstruction and generation comparisons, and its transformer is reused as the denoiser architecture.","marker":"[19]"},{"why":"MAGVIT-v2 provides the 3D causal CNN encoder-decoder backbone and the 2D-to-3D weight inflation strategy used to train the factorized autoencoder.","marker":"[58]"},{"why":"PVDM introduces the tri-plane latent video representation that the four-plane design extends and is compared against as a prior work.","marker":"[59]"},{"why":"HVDM is the hybrid tri-plane/wavelet video diffusion baseline in the class-conditional generation comparison.","marker":"[30]"},{"why":"VIDIM is the pixel-space diffusion interpolation baseline that the interpolation experiments in Table 3 are measured against.","marker":"[27]"},{"why":"UCF-101 is the dataset for class-conditional generation and interpolation evaluation, where all FVD comparisons are reported.","marker":"[47]"},{"why":"Kinetics-600 is the dataset used to train the autoencoders and for the frame-prediction evaluation.","marker":"[6]"},{"why":"DDPM supplies the diffusion training objective and sampler used for the latent diffusion models.","marker":"[21]"},{"why":"FVD is the evaluation metric used to compare generation quality across all tasks.","marker":"[48]"}],"fun_headline_variants":["Video latents factorized onto four planes, halving diffusion sequence","Four-plane video factorization shrinks diffusion tokens from 1280 to 672","Factorized video autoencoder: four planes replace volume latents for speed","Video diffusion gets faster with four-plane latent factorization","Efficient video modelling via four-plane factorization of volumetric latents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The re-trained W.A.L.T. baseline (W.A.L.T.*) faithfully represents the original W.A.L.T. model, even though its FVD differs from the published number (39 vs 46 on UCF-101 128×128).","fun_headline_variants_meta":{"raw":{"variants":["Video latents factorized onto four planes, halving diffusion sequence","Four-plane video factorization shrinks diffusion tokens from 1280 to 672","Factorized video autoencoder: four planes replace volume latents for speed","Video diffusion gets faster with four-plane latent factorization","Efficient video modelling via four-plane factorization of volumetric latents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1346,"prompt_tokens":917,"completion_tokens":429,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":340}},"tokens_in":533,"tokens_out":429,"duration_ms":5098,"temperature":1.0,"reasoning_tokens":340,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:24:37.977861+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same transformer diffusion model on the volumetric latent with the same data and compute, and measure FVD on UCF-101 128×128: if the volumetric model matches or beats the factorized model's 38—or if the original W.A.L.T.'s published 46 is reproduced instead of the 39 reported for W.A.L.T.*—then the claim that four-plane factorization preserves generation quality is undermined. The paper's own Table 4 offers a second check: swapping mean pooling for linear projection worsens FVD to 50, showing the result depends on that design choice.","supporting_citations":[{"cited_title":"Photorealistic video generation with diffusion models, 2023","cited_arxiv_id":null,"evidence_quote":"The W.A.L.T. model is the volumetric baseline for reconstruction and generation comparisons, and its transformer is reused as the denoiser architecture."},{"cited_title":"Language model beats diffusion - tokenizer is key to visual generation","cited_arxiv_id":null,"evidence_quote":"MAGVIT-v2 provides the 3D causal CNN encoder-decoder backbone and the 2D-to-3D weight inflation strategy used to train the factorized autoencoder."},{"cited_title":"Hybrid video diffusion models with 2d triplane and 3d wavelet rep- resentation","cited_arxiv_id":null,"evidence_quote":"HVDM is the hybrid tri-plane/wavelet video diffusion baseline in the class-conditional generation comparison."},{"cited_title":"Video inter- polation with diffusion models","cited_arxiv_id":null,"evidence_quote":"VIDIM is the pixel-space diffusion interpolation baseline that the interpolation experiments in Table 3 are measured against."},{"cited_title":"Denoising dif- fusion probabilistic models","cited_arxiv_id":null,"evidence_quote":"DDPM supplies the diffusion training objective and sampler used for the latent diffusion models."}],"review_version":1}