{"id":"99c1963e-5516-4a4d-b5fc-59683d050dd6","arxiv_id":"2411.16375","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"By making temporal attention causal and sharing cached key-value features across all denoising steps, Ca2-VDM cuts autoregressive video generation cost from quadratic to roughly linear in the number of generated chunks.","lead":"Ca2-VDM speeds up autoregressive video generation by letting the model reuse cached features of earlier frames instead of recomputing them at every step. It reports comparable video quality to prior systems while cutting computation and memory for long videos.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The efficiency mechanism is well-supported, but the quality half of the central claim depends on causal fine-tuning of bidirectional pretrained temporal attention, which Appendix F admits may be sub-optimal; no controlled T2V comparison against a bidirectional extendable-condition baseline is…","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the quality half of the central claim depends on replacing pretrained bidirectional temporal attention with causal attention via fine-tuning, a step the paper itself flags as potentially sub-optimal in Appendix F. I do not see a fatal flaw in the cache-sharing mechanism itself; the training objective in Eq. (2), the per-frame timestep embedding tEmb(0) for clean prefixes, and the Cyclic-TPEs scheme are internally consistent with the reported inference procedure. The efficiency evidence, including accumulated time, FLOPs breakdown, and memory scaling in Table 6, independently supports the speedup. What remains unproven is that quality stays comparable to a bidirectional extendable-condition baseline on the headline text-to-video task. The only such baseline trained by the authors, OS-Ext, is evaluated on SkyTimelapse only, and the VBench numbers there show small but unreplicated gaps on temporal metrics. A controlled OS-Ext training run on InternVid with identical budget and condition length would settle whether causal fine-tuning, rather than the larger context or extra training stages, is the cause of any observed quality difference. Since the reader already issued a CONDITIONAL verdict based on this same concern, my stress-test does not move the verdict; it sharpens the required check.","tokens_in":20892,"tokens_out":13159,"duration_ms":123133,"concrete_test":"Train OS-Ext (bidirectional, extendable condition) on the same filtered InternVid subset with identical Pmax=49, l=16, resolution, and training-step budget as Ca2-VDM, then evaluate MSR-VTT and UCF101 FVD plus the four VBench metrics with at least three seeds. If the bidirectional baseline's mean scores exceed Ca2-VDM's by more than the pooled seed standard error, the causal-mask fine-tuning is the likely cause of the quality gap, and 'comparable quality' should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Ca2-VDM's speed and memory claims are credible: t=0-conditioned clean-prefix KVs are causally computable, shared across denoising steps, and the queue bounds memory. But the central claim is a conjunction: speedup plus 'comparable' quality. The quality conjunct rests on Section 4.1's initialization from Open-Sora, whose temporal attention is bidirectional, with a causal mask and per-frame timestep embeddings imposed only during fine-tuning. Appendix F explicitly concedes that fine-tuning these weights into causal attention 'might be sub-optimal' and that no causal pretraining was performed. The quality evidence is not yet sufficient to rule out a real degradation: Table 1 mixes condition types and training recipes across baselines; the only bidirectional extendable-condition baseline trained by the authors is OS-Ext, and it is trained only on SkyTimelapse, not on the InternVid text-to-video task. Table 7 reports Ca2-VDM below OS-Ext on motion smoothness (97.59 vs 98.93) and temporal flickering (97.14 vs 98.57) with no error bars, and Tables 1-4 report no seed variance. Because Ca2-VDM uses a larger condition length (Pmax=49) and a two-stage training recipe than OS-Fix (P=16), part of its apparent quality parity could come from these confounds rather than from the causal cache design. If causal fine-tuning actually costs a few FVD or VBench points, the efficiency result still stands but the 'comparable quality' half of the central claim does not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Ca2-VDM, an autoregressive video diffusion transformer that replaces bidirectional temporal attention with causal temporal attention and caches keys/values of clean conditional frames, sharing them across all denoising steps and storing them in a bounded queue with cyclic temporal positional embeddings. The claimed effect is that autoregressive generation cost grows roughly linearly rather than quadratically in the number of autoregression steps, that GPU memory is independent of the number of denoising steps, and that generation quality remains comparable to bidirectional baselines. The efficiency claims are supported by timing, FLOPs, and memory measurements that match the algorithmic design; the quality claim is only partially supported, with the main controlled comparison against a bidirectional extendable-condition baseline limited to the SkyTimelapse video-prediction task.","tokens_in":21159,"tokens_out":5805,"duration_ms":53347,"significance":"If the efficiency results hold, this is a useful contribution: the KV-cache queue for video diffusion is not a trivial extension of language-model KV-caching because the diffusion model is called repeatedly with different timesteps, and the paper correctly identifies the two obstacles of cache computation and cache storage and proposes concrete solutions. The paper provides measured time, FLOPs, and memory numbers, the complexity analysis matches the algorithm, and the code is released. The quality evidence is currently the weak part of the central claim: Appendix F explicitly concedes that fine-tuning bidirectionally pretrained Open-Sora weights into causal attention might be sub-optimal, and the only bidirectional extendable-condition baseline trained by the authors, OS-Ext, is evaluated only on SkyTimelapse. With an additional controlled text-to-video comparison against a bidirectional extendable-condition baseline, or with a claim scoped to efficiency only, the paper would be substantially stronger.","major_comments":[{"comment":"The quality half of the central claim, that quality remains comparable to bidirectional models, is not established by the current experiments. Table 7 shows Ca2-VDM below OS-Ext on motion smoothness (97.59 vs 98.93) and temporal flickering (97.14 vs 98.57), with no error bars, and this is the only direct bidirectional extendable-condition baseline trained by the authors. Appendix F additionally concedes that fine-tuning bidirectional Open-Sora weights into causal attention 'might be sub-optimal' because no causal pretraining was performed. Because the speedup is real even if quality degrades, this point is load-bearing for the full claim. Please add a controlled text-to-video comparison against an OS-Ext-style bidirectional baseline on the InternVid/MSR-VTT task, and report per-seed variance for the quality tables.","section":"§4.2, Table 7, and Appendix F"},{"comment":"The zero-shot FVD comparisons in Table 1 are not apples-to-apples. Ca2-VDM is listed with condition T+I, meaning it receives the ground-truth first frame as an image condition, while several baselines (ModelScope, VideoComposer, Make-A-Video) are text-only; on MSR-VTT, Ca2-VDM matches SEINE, which is also a T+I method. Reporting a T+I model alongside text-only models under the heading 'zero-shot text-to-video' conflates the advantage of the extra condition with generation quality. The text should either restrict the state-of-the-art/comparable claim to T+I models or provide a text-only Ca2-VDM variant.","section":"§4.2, Table 1"},{"comment":"The temporal-consistency evaluation in Table 3 is described inconsistently: the caption says 'FVD between AR step 1 and i', while §D.2 says the FVD of three 16-frame chunks is computed 'w.r.t. the 16-frame ground-truth videos'. These are different quantities, and the surrounding text claims lower FVD indicates better consistency without stating which reference distribution is used. Please clarify the metric and the reference distribution; if the numbers are chunk-wise FVD to ground truth, the caption should say so, and the claim that Ca2-VDM has lower FVD than OS-Fix and StreamT2V should be tied to the correct comparison.","section":"§3.3, Table 3, and §D.2"}],"minor_comments":[{"comment":"The abstract claims 'state-of-the-art quantitative and qualitative video generation results', while the conclusion says the model 'shows comparable generation quality with existing SOTA VDMs'; these two claims should be aligned.","section":"Abstract and §5"},{"comment":"The ablation text says 'generated 48 frames by 6 AR steps' for Table 4, but §D.2 says 'each model generated 96 frames with 6 AR steps and l=16'; with l=16, six AR steps produce 96 frames. Please correct the inconsistent number.","section":"§4.2 and §D.2, Table 4"},{"comment":"Quality tables do not report the number of generated videos used for each FVD/VBench estimate or standard deviations; adding these would materially help assess the small quality differences reported.","section":"Tables 3, 4, and 7"},{"comment":"The text contains 'All most all Existing VDMs', which should read 'Almost all existing VDMs', and the footnote marker after 'Cyclic-TPEs 1.' is orphaned.","section":"§1 and footnote 1"},{"comment":"The definition of the sub-prefix indices in Eq. (4) is terse; please clarify that h^{P-P'}_0 through h^{P-1}_0 are the clean-prefix hidden states at the current layer, and that the broadcasting for i < P is over the spatial dimension.","section":"Eq. (4)"}],"recommendation":"major_revision","confidential_remarks":"The efficiency contribution is solid and likely publishable, but the quality claim needs a controlled bidirectional baseline on the text-to-video task. If the authors add that comparison, or reduce the central claim to efficiency only, the paper could be accepted after revision; as is, the abstract overstates the quality result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The efficiency mechanism is the real story here, and it is a good one. The paper's two pieces—causal temporal attention so prefix features can be computed once and reused, and sharing one t=0-conditioned KV cache across all denoising steps—are genuinely new relative to the cited literature. Live2diff also uses unidirectional attention but keeps per-step caches; sharing one cache across timesteps via a clean-prefix timestep embedding is the actual contribution, and it is not a trivial extension of LLM KV caching because of the repeated model calls during denoising. The time, FLOPs, and memory measurements all match the algorithmic design: the queue bounds memory regardless of denoising steps, and the cost grows roughly linearly instead of quadratically in autoregression length. That part is credible and well evidenced.\n\nThe soft spot is the quality half of the central claim. The abstract says state-of-the-art, but the body and Conclusion say comparable, and the comparability evidence has holes. The model is initialized from Open-Sora v1.0, whose temporal attention was trained bidirectionally, and Appendix F explicitly concedes that fine-tuning these weights into causal attention \"might be sub-optimal\" and that no causal pretraining was performed. The VBench numbers in Table 7 put Ca2-VDM below OS-Ext on motion smoothness and temporal flickering, with no error bars, and Tables 1-4 report no seed variance. More importantly, the only bidirectional extendable-condition baseline trained on the T2V task is not present: OS-Ext was trained only on SkyTimelapse, so the main text-to-video comparison against a bidirectional baseline of comparable training budget is missing. That leaves a plausible source of quality degradation unquantified.\n\nNone of this undermines the efficiency result, and the paper is honest about its limitations. The circularity burden is low: the speedup is measured, not fitted, and the OS-Fix/OS-Ext baselines are appropriate controls for efficiency. The citation pattern is fair, including the concurrent Live2diff. Reproducibility is partial—code is available, but training data are not fully shipped—and the appendix reports the important details.\n\nWho is this for? Anyone working on long or streaming video generation will want the cache-sharing idea, and the paper should get a serious referee. The referee should push on the quality evaluation: a controlled T2V comparison against OS-Ext on InternVid, seed variance, and ideally a study of how much the bidirectional-to-causal fine-tuning gap actually costs. As-is, I would accept the efficiency claim and treat the quality claim as promising but unproven.","headline":"Solid efficiency contribution with measured speedups and an honest appendix, but the 'comparable quality' half of the central claim is under-supported by the current experiments.","tokens_in":21764,"tokens_out":1135,"would_cite":true,"duration_ms":12817,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By making temporal attention causal, video diffusion models can cache conditioning frames once and reuse them at every denoising step, cutting autoregressive generation cost from quadratic to roughly linear.","keywords":["video diffusion models","autoregressive generation","KV-cache","causal temporal attention","long-term video generation","inference acceleration","cache sharing","latent diffusion"],"falsifier":"Generate identical prompts and starting frames with a causal-fine-tuned model and with the same model restored to bidirectional attention, and compare FVD chunk by chunk; if the causal variant is materially worse on early chunks where the bidirectional model is strong, the central quality claim fails. A sharper probe: test whether recomputing the clean prefix features at every denoising timestep instead of using the shared $t=0$ cache changes the output; if the videos differ noticeably, the cache-sharing premise is violated.","tokens_in":22,"feed_emoji":"🎬","tokens_out":6707,"duration_ms":120934,"temperature":0.7,"pith_summary":"Autoregressive video diffusion models waste compute by re-encoding the same conditioning frames at every generation step, and that waste grows quadratically when the condition is extended to carry long-term context. This paper claims that replacing the bidirectional temporal attention in a video diffusion transformer with causal temporal attention makes those conditioning features cacheable: keys and values of the clean prefix frames are computed once and reused in every subsequent step. A second mechanism, cache sharing, gives the clean prefix its own timestep embedding $t=0$, so the same cache is valid at every denoising timestep and GPU memory no longer grows with the number of denoising steps. If the paper is right, arbitrary-length video generation can run at roughly linear cost with quality comparable to bidirectional models.","feed_headline":"KV caching makes autoregressive video generation linear, not quadratic","feed_subtitle":"Reusing cached condition frames across denoising steps cuts compute and GPU memory at comparable quality.","key_machinery":"The load-bearing object is a causal temporal attention mask: a lower-triangular mask $M$ in $\\mathrm{Softmax}(QK^\\top/\\sqrt{C'} + M)$ forces every frame to attend only to its predecessors. That unidirectional computation makes it possible to maintain a temporal KV-cache queue: clean conditional frames' keys and values are written once at timestep $t=0$ and reused by every later denoising stage. Cache sharing across denoising steps is justified by giving the clean prefix its own timestep embedding $t=0$ in both training and inference, so the cached features never depend on the noise level of the target frames. Two supporting mechanisms make the pipeline practical: prefix-enhanced spatial attention concatenates a short clean sub-prefix into the spatial attention keys and values to strengthen conditioning, and cyclic temporal positional embeddings keep positional information aligned with cached keys and values once generation exceeds the training length.","core_discovery":"On its own terms, the paper's discovery is that unidirectionality plus a timestep-agnostic condition feature is enough to remove almost all redundant autoregressive computation from video diffusion. With causal temporal attention, a frame's features depend only on earlier frames, so the keys and values of the already-generated clean frames can be precomputed in a 'cache writing' pass at $t=0$ and then read, not recomputed, during each denoising step of each later autoregression step. The same cache is shared across all denoising timesteps because the clean prefix is always embedded with $t=0$ rather than by the current noise-level timestep. The paper reports that this design reaches FVD scores similar to bidirectional baselines while generating 80 frames in about half the time of a fixed-condition baseline and at constant KV-cache memory independent of denoising steps.","pith_inferences":["If the authors' caveat is right that no causal pretraining was performed, training a video diffusion transformer with causally masked temporal attention from scratch could close the remaining quality gap to bidirectional models while keeping the same cache benefits.","The same cache-sharing design should transfer to other iterative generative models that denoise in many steps while conditioned on fixed context, such as image-to-video, audio, or long-horizon world models, and to samplers with variable step counts at no extra cache memory.","The cyclic positional-embedding mechanism implicitly assumes scene statistics repeat with a period equal to the training window; very long generations beyond that window are a natural stress test of whether periodic position assignment introduces visible looping artifacts."],"forward_implications":["Autoregressive video generation with extendable conditioning becomes roughly linear in compute per generated chunk rather than quadratic, because conditional frames are never re-encoded.","The KV-cache queue bounds memory independently of the number of denoising steps, so longer sampling schedules or more denoising steps no longer multiply cache cost.","Increasing the maximum condition length $P_{\\max}$ improves temporal consistency for a small additional cost, since extended context enters only as cached keys and values.","The causal conversion with cache sharing applies to both text-to-video and video prediction and reaches FVD scores comparable to bidirectional baselines in the reported settings."],"supporting_citations":[{"why":"Provides the pretrained video diffusion transformer whose bidirectional temporal attention is converted to causal attention.","marker":"(Zheng et al., 2024)"},{"why":"Defines the spatial-temporal transformer block structure that the proposed model is built on.","marker":"(Ma et al., 2025)"},{"why":"Supplies the text-conditioning and training recipe adopted for text-to-video generation.","marker":"(Chen et al., 2024a)"},{"why":"Establishes the diffusion objective and noise-schedule reparameterization used in training.","marker":"(Ho et al., 2020)"},{"why":"Provides the improved DDPM inference schedule and learnable covariance objective used at sampling.","marker":"(Nichol & Dhariwal, 2021)"},{"why":"Is the concurrent KV-cache autoregressive VDM used for GPU-memory comparison, showing the benefit of sharing caches across denoising steps.","marker":"(Xing et al., 2024)"},{"why":"Is the autoregressive baseline compared on time cost and temporal consistency.","marker":"(Henschel et al., 2025)"}],"fun_headline_variants":["Autoregressive video diffusion goes linear with cache sharing","Causal generation + cache reuse speeds up video diffusion","Ca2-VDM: linear-time autoregressive video generation","Sharing caches across denoising steps cuts video diffusion compute","Unidirectional features make video diffusion cache-friendly"],"cache_read_input_tokens":23808,"weakest_assumption_plain":"The assumption the argument rests on is that converting a pretrained bidirectional video diffusion transformer's temporal attention into causal attention, with only fine-tuning, does not meaningfully degrade generation quality.","fun_headline_variants_meta":{"raw":{"variants":["Autoregressive video diffusion goes linear with cache sharing","Causal generation + cache reuse speeds up video diffusion","Ca2-VDM: linear-time autoregressive video generation","Sharing caches across denoising steps cuts video diffusion compute","Unidirectional features make video diffusion cache-friendly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1423,"prompt_tokens":972,"completion_tokens":451,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":373}},"tokens_in":588,"tokens_out":451,"duration_ms":5229,"temperature":1.0,"reasoning_tokens":373,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:11:48.504295+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate identical prompts and starting frames with a causal-fine-tuned model and with the same model restored to bidirectional attention, and compare FVD chunk by chunk; if the causal variant is materially worse on early chunks where the bidirectional model is strong, the central quality claim fails. A sharper probe: test whether recomputing the clean prefix features at every denoising timestep instead of using the shared $t=0$ cache changes the output; if the videos differ noticeably, the cache-sharing premise is violated.","supporting_citations":[],"review_version":1}