{"id":"9e28c3c8-b53d-4271-a1bc-3c4d2380c3ee","arxiv_id":"2511.12940","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Augmenting a diffusion video transformer with an RNN memory block and frame-wise overlapping attention improves long-horizon consistency, with simple LSTM matching newer Mamba2 and TTT memory blocks.","lead":"This paper adds a recurrent memory module (LSTM, Mamba2, or TTT) to a video-diffusion transformer, so the model can remember frames older than its attention window. It generates video frame by frame with overlapping attention and a training trick called hidden-state pre-fetching, and reports better long-video consistency on Maze and Minecraft benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'no training and inference gap' claim is unsupported: training pre-fetches hidden states from ground-truth frames, but inference updates them from model-generated frames; no experiment quantifies this shift.","rationale":"The reader's weakest assumption matches the most load-bearing issue I can identify. The paper's central 'no training and inference gap' claim is the crux: if the gap exists, the method is a standard free-running recurrent diffusion model with a clever training shortcut, not a fundamentally gap-free formulation. The manuscript itself flags the assumption in §4.3 ('Assuming the previous hidden state stacks sufficient context information...') and then in §4.2 attempts to hand-wave the inference side by calling final-step denoised outputs 'clean frames.' Those are not the same as the ground-truth frames used in prefetch. The §6.2 ablation demonstrates memory sensitivity to input noise, which makes the unmeasured generated-input drift more concerning. I would not reject the paper: the frame-wise vs chunk-wise improvement is an interesting empirical finding, and the prefetch trick is a useful engineering contribution. But the current text overclaims. A single oracle-memory experiment would settle whether the gap is material; until then, conditional is the right verdict. No change to the reader's verdict.","tokens_in":15296,"tokens_out":5716,"duration_ms":54310,"concrete_test":"Run RAD-LSTM-f on the Maze evaluation set with two inference memory conditions: (A) as described in the paper, update RNN hidden states from the model's own final-DDIM-step predictions; (B) oracle memory, update hidden states from ground-truth clean frames at the same time points while keeping all other generation steps identical. Compare PSNR/SSIM/LPIPS per sequence and per frame over the predicted 40%. If (B) materially outperforms (A) (e.g., mean SSIM difference > 0.02, or > the seed-to-seed variation from 3 seeds), the training/inference gap is real and the central claim should be relaxed; if (A) ≈ (B), the concern is resolved. Also report the L2 drift between hidden states under (A) and (B) over the first 100 predicted frames.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on hidden states seen during training and inference being equivalent. In §4.3, hidden states are pre-fetched by running the RNN over clean ground-truth frames with window size 1, and the text asserts these are 'equivalent to the standard DiT with RNN by sliding-window.' At inference (§4.2), no ground-truth frames exist: the RNN is updated only at the final DDIM step using the model's own generated latents. The phrase 'all memory inputs consist of clean frames' conflates 'denoised' with 'ground truth.' Generated latents carry sampling error and distribution shift, and on long horizons this shift accumulates as memory is recursively updated. No experiment measures this shift. §6.2 ablates noised vs. clean memory frames at training, which is a different manipulation and, if anything, shows memory quality is sensitive to the input frame distribution. The chunk-wise variant may have procedural train/inference consistency, but the frame-wise variant—the best-performing configuration—does not. Thus the headline 'no training and inference gap' is unsupported and appears internally inconsistent with the described inference loop.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RAD, a recurrent diffusion transformer for long video generation. RAD augments a DiT-based video diffusion model with recurrent memory blocks (LSTM, Mamba2, TTT), compares chunk-wise and frame-wise autoregressive training paradigms, and introduces a hidden-state pre-fetching scheme that allows attention computation to be parallelized during training while retaining recurrent memory. Experiments are reported on Memory Maze and Minecraft datasets, with the main empirical claims that (i) LSTM is competitive with or better than Mamba2 and TTT in this setting, (ii) frame-wise autoregression with overlapping windows substantially improves over chunk-wise autoregression, and (iii) the proposed training procedure eliminates the training/inference gap typical of recurrent diffusion models.","tokens_in":15604,"tokens_out":3972,"duration_ms":37565,"significance":"If the results hold, the paper makes several useful contributions to video world modeling: a controlled comparison of RNN memory blocks inside a common DiT backbone, an analysis of chunk-wise versus frame-wise autoregressive memory update, and a concrete parallelization strategy for RNN-augmented attention training. The hyperparameter table (Table 5), the efficiency analysis (Table 3), and the ablations on action conditioning (Table 4) and memory-frame noise (§6.2) are helpful and contribute to reproducibility. However, the central headline claim of 'no training and inference gap' is not established by the reported experiments, and the quantitative comparisons lack error bars or multiple seeds, so the significance of the main empirical differences cannot yet be assessed.","major_comments":[{"comment":"The claim of 'no training and inference gap' is unsupported. In training (§4.3), hidden states are pre-fetched from ground-truth clean frames with window size 1; in inference (§4.2), hidden states are updated from the model's own generated frames at the final DDIM step. The text in §4.2 states that this 'ensur[es] that all memory inputs consist of clean frames', but this equates 'clean' in the sense of 'denoised' with 'ground-truth'. Generated frames carry sampling error and can drift on long horizons, so the recurrent memory distribution at inference will differ from that seen during training. The assertion in §4.3 that 'the pre-fetched hidden states are equivalent to the standard DiT with RNN by sliding-window' is an assumption, not a theorem or a measurement. The ablation in §6.2 compares noised versus clean memory frames during training, which is a different manipulation and does not","section":"§4.2, §4.3, Abstract"},{"comment":"All quantitative comparisons are reported as single runs without error bars, seeds, or significance tests. Several of the conclusions rest on small differences: in Table 2, the frame-wise PSNR values are 16.70 (Mamba2), 16.72 (TTT), and 16.59 (LSTM); in Table 1, frame-wise PSNR values are 15.35, 15.50, and 15.30. The paper claims that frame-wise variants perform similarly and that LSTM is superior in the chunk-wise setting, but without variance estimates these claims cannot be evaluated. Please report results across at least three seeds with mean and standard deviation, or provide statistical tests, for both datasets and all key configurations.","section":"Tables 1 and 2"},{"comment":"The hidden-state pre-fetching mechanism is a central component of the training procedure, but its correctness rests on an unverified assumption. The text reads: 'Assuming the previous hidden state stacks sufficient context information, the pre-fetched hidden states are equivalent to the standard DiT with RNN by sliding-window.' This assumption is load-bearing: if pre-fetched states from clean ground-truth frames are not representative of the states the model would produce when processing its own outputs, then the training objective optimizes a different distribution from the inference procedure, and the claimed parallel-attention efficiency gain comes at the cost of a train/inference mismatch. The paper does not report any diagnostic that checks this equivalence, such as comparing pre-fetched hidden states with online-computed hidden states on the same clean training frames, or measuring","section":"§4.3"}],"minor_comments":[{"comment":"Typo: 'bottlenece' should be 'bottleneck'.","section":"§4.3"},{"comment":"The LSTM equations use y_{t-1} and y_t without clearly defining the relation between y_t and the hidden state h_t. This is potentially confusing because the general RNN form at the top defines h_t and y_t separately, but the LSTM update seems to set y_t as the output. Please clarify the notation, especially since the paper later refers to 'hidden state h_t' and 'memory' interchangeably.","section":"§3.1"},{"comment":"The text refers to 'Table 7b', but there is no Table 7 in the main text. The ablation results in §6.2 should be presented in a properly numbered table or figure with a clear caption.","section":"§6.2"},{"comment":"The evaluation protocol is underspecified. For both datasets, please state whether the reported PSNR/SSIM/LPIPS are averaged over all predicted frames or over a fixed horizon, how the ground-truth comparison is aligned with the generated frames, and how many evaluation videos are used in each reported number.","section":"§5.1"},{"comment":"The terms 'chunk-wise' and 'frame-wise' are defined informally in the text and Figure 3, but the exact chunk size and overlap are not given in Table 5. In particular, the attention window size is listed as 20 for both training and inference, but the chunk size used in chunk-wise experiments is not reported. Please add the chunk size and the sliding step to the hyperparameter table.","section":"§4.2, Table 5"},{"comment":"The right panel labels 'Hidden state pre-fetching' and 'Diffusion forward' are difficult to read and the diagram is cluttered. Consider separating the two stages into distinct subfigures and enlarging the text.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper's novelty relative to prior work (especially Diffusion Forcing [10] and Po et al. [35]) should be sharpened. The frame-wise autoregressive scheme with overlapping windows is closely related to Diffusion Forcing, and the idea of using clean-frame hidden states for memory is similar in spirit to Po et al.'s strategy of leaving initial frames clean, though the present paper extends it to all frames. The revision should clearly articulate what is new beyond these existing components. The lack of error bars is a serious issue for a systems paper whose comparisons are mostly close; I would like the editor to require multiple seeds before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RNN plus overlapping window attention is a reasonable engineering direction, and the prefetch trick for parallel training is the most useful idea in the paper. But the 'no training and inference gap' claim is exactly the kind of thing that needs a direct test, and it doesn't get one.\n\nWhat's new: the specific combination of an RNN block after attention in a DiT, frame-wise autoregressive rollout with overlapping sliding windows, and a hidden-state prefetch scheme that decouples the RNN from attention during training. The systematic comparison of LSTM, Mamba2, and TTT in one architecture is a genuine contribution, and the finding that LSTM holds its own against much newer blocks is worth reporting. The frame-wise vs chunk-wise ablation is also informative: overlapping windows shift the information burden from the hidden state to attention, which explains why architecture choice matters less in the frame-wise regime. That is a real insight.\n\nSoft spots: the paper's headline claim is unsupported as written. Training pre-fetches hidden states from ground-truth clean frames (Section 4.3). Inference updates the hidden state from the model's own generated frames at the final DDIM step (Section 4.2). Those are different distributions. The assertion that the pre-fetched states are 'equivalent' is an assumption, and Section 6.2's own ablation—noised versus clean memory frames—shows that memory quality is sensitive to the input distribution. That is direct evidence the gap is not just theoretical. The fix is straightforward: either run the RNN on generated frames during training, or at least measure the divergence between hidden states from real and generated frames. Without that, 'no training and inference gap' is an overclaim, not a result.\n\nAlso, Tables 1 and 2 show single runs with no error bars or seeds. Several differences between variants (e.g., LSTM-f vs Mamba2-f on Maze, 15.30 vs 15.35 PSNR) are around 0.05, which is likely noise. The chunk-wise setting actually degrades performance on Minecraft relative to the DF baseline, which complicates the 'global memory helps' narrative—the memory seems to help mainly when attention already has overlap. Those are minor if the no-gap claim gets fixed, but they should be addressed.\n\nWho this is for: researchers working on long-horizon video diffusion world models, especially autoregressive rollout and recurrent memory. The prefetch trick is worth knowing even if the headline claim fails.\n\nRecommendation: send to peer review. The architecture and the comparison are solid and worth referee time. But the editor should push for a direct test of the gap claim; the paper is conditional until then.","headline":"RNN plus overlapping-window attention is a solid engineering contribution, but the 'no training and inference gap' claim needs a direct test and currently overreaches.","tokens_in":16090,"tokens_out":2410,"would_cite":true,"duration_ms":21052,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes Recurrent Autoregressive Diffusion (RAD), which augments each diffusion transformer layer with an LSTM memory block and generates frames with overlapping sliding-window attention, claiming this removes the training-infere","keywords":["autoregressive video generation","diffusion transformer","recurrent memory","LSTM","long-horizon consistency","hidden-state prefetch","sliding-window attention","world models"],"falsifier":"A direct test would be to run a trained frame-wise RAD model autoregressively for many more steps than the training sequences, then compare the distribution of its hidden states (or a probe such as the SSIM of the first generated frame conditioned purely on memory) against hidden states computed from ground-truth frames. If the generated-state memory diverges measurably—for example, the cell-state norm or reconstruction of a held-out frame degrades with rollout length—the clean-frame prefetch equivalence is violated. A simpler version: train with noised-frame memory (the paper's own ablation)","tokens_in":15197,"feed_emoji":"🧠","tokens_out":4237,"duration_ms":34494,"temperature":0.7,"pith_summary":"The paper proposes Recurrent Autoregressive Diffusion (RAD), a video-generation architecture that adds a recurrent neural network (an LSTM) to each layer of a diffusion transformer. The RNN compresses history beyond the attention window into a fixed-size hidden state, while overlapping sliding-window attention preserves local detail. RAD's central claim is that this combination removes the usual training-inference gap in recurrent diffusion models: hidden states are computed from clean frames during training (hidden-state prefetching) and from freshly generated clean frames during inference, both with the same frame-wise protocol. If correct, long-horizon video generation can stay globally consistent with bounded memory. The paper also reports that a simple LSTM matches or beats more recent recurrent blocks (Mamba2 and TTT) in this setting.","feed_headline":"LSTM memory keeps long videos stable across thousands of frames","feed_subtitle":"Fixed-size RNN memory plus overlapping local attention keeps long video generations consistent.","key_machinery":"The central mechanism is the RNN memory block inserted after attention in each DiT layer, combined with a hidden-state prefetching procedure. During training, hidden states are precomputed by running the RNN with a window size of 1 over clean frames, so all attention computations can be done in parallel; at inference the same frame-wise sliding-window protocol is used, with hidden states updated only after the final denoising step so the memory always sees clean frames. This alignment of training and inference procedures is what the paper claims removes the train/test gap.","core_discovery":"The paper's central claim is that a diffusion transformer augmented with an RNN memory block can generate arbitrarily long videos if the autoregressive rollout is frame-wise rather than chunk-wise, with full attention over overlapping sliding windows. The discovery is twofold: (1) with chunk-wise autoregression, where hidden states alone must bridge non-overlapping windows, LSTM's explicit separation of short-term output and long-term cell state makes it the strongest recurrent choice, outperforming Mamba2 and TTT; (2) in frame-wise mode, all three recurrent blocks perform comparably because attention carries local pixel information, letting hidden states specialize in global memory. The hea","pith_inferences":["A natural extension of the paper's clean-frame memory design is to test robustness to distribution drift by training with the model's own denoised frames instead of ground-truth frames, directly probing the assumed equivalence that the prefetch relies on.","Because frame-wise attention erases architectural differences among RNNs, a cost-conscious follow-up could replace the LSTM with a much cheaper linear recurrence (or a learned coordinate) once local continuity is guaranteed by attention.","The chunk-wise result, where LSTM thrives only when memory must carry pixel detail, suggests that recurrent architectures with separate channels for local and global state are preferable whenever overlap-free windows are unavoidable, which could inform non-video sequence-modeling tasks.","The prefetch mechanism decouples memory updates from diffusion-noise timesteps, so memory could in principle be updated asynchronously or at lower frequency during inference, a practical optimization the paper does not explore."],"forward_implications":["Long-video world models no longer need to choose between bounded memory and long-range consistency: a fixed-size LSTM hidden state can retain history beyond the attention window while attention preserves local detail.","Recurrent memory in diffusion transformers can be trained as efficiently as standard DiT, because hidden-state prefetching restores parallelism in attention computation.","A simple LSTM is a viable default memory block for autoregressive video diffusion, matching or exceeding more recent recurrent architectures like Mamba2 and TTT.","Frame-wise autoregression with overlapping windows is more important for generation quality than the choice of recurrent architecture, since local attention relieves the memory of carrying pixel-level detail.","The no-training-inference-gap design, with clean frames feeding memory at both stages, makes long-horizon generation stable enough to evaluate on challenging benchmarks like Memory Maze and Minecraft."],"fun_headline_variants":["LSTM memory keeps long video generation consistent","Global memory and local attention for ultra-long videos","Recurrent autoregressive diffusion: no more video forgetting","Frame-wise diffusion with LSTM outperforms chunk-wise","Diffusion transformer with recurrent memory for infinite video"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that hidden states computed from ground-truth clean frames during training are representative of the hidden states the model will see from its own generated frames at inference; if the model's outputs drift, the memory becomes corrupted and the claimed 'no training and inference gap' fails.","fun_headline_variants_meta":{"raw":{"variants":["LSTM memory keeps long video generation consistent","Global memory and local attention for ultra-long videos","Recurrent autoregressive diffusion: no more video forgetting","Frame-wise diffusion with LSTM outperforms chunk-wise","Diffusion transformer with recurrent memory for infinite video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1178,"prompt_tokens":720,"completion_tokens":458,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":386}},"tokens_in":464,"tokens_out":458,"duration_ms":4311,"temperature":1.0,"reasoning_tokens":386,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T21:55:48.380846+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to run a trained frame-wise RAD model autoregressively for many more steps than the training sequences, then compare the distribution of its hidden states (or a probe such as the SSIM of the first generated frame conditioned purely on memory) against hidden states computed from ground-truth frames. If the generated-state memory diverges measurably—for example, the cell-state norm or reconstruction of a held-out frame degrades with rollout length—the clean-frame prefetch equivalence is violated. A simpler version: train with noised-frame memory (the paper's own ablation)","supporting_citations":[],"review_version":1}