Pith. sign in

REVIEW 3 major objections 6 minor 4 cited by

Recurrent Autoregressive Diffusion: Global Memory Meets Local Attention

T0 review · 3 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes Recurrent Autoregressive Diffusion (RAD), which augments each diffusion transformer layer with an LSTM memory block and generates frames with overlapping sliding-window attention, claiming this removes the training-infere

desk verdict RNN plus overlapping-window attention is a solid engineering contribution, but the 'no training and inference gap' claim needs a direct test and currently overreaches. read the letter →

arxiv 2511.12940 v2 pith:5PKMDXMB submitted 2025-11-17 cs.CV

Taiye Chen , Zihan Ding , Anjian Li , Christina Zhang , Zeqi Xiao , Yisen Wang , Chi Jin This is my paper · ORCID
classification cs.CV
keywords autoregressivevideogenerationdiffusiontransformerrecurrentmemoryLSTMlong-horizonconsistencyhidden-stateprefetchsliding-windowattentionworldmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Recurrent Autoregressive Diffusion (RAD), a video-generation architecture that adds a recurrent neural network (an LSTM) to each layer of a diffusion transformer. The RNN compresses history beyond the attention window into a fixed-size hidden state, while overlapping sliding-window attention preserves local detail. RAD's central claim is that this combination removes the usual training-inference gap in recurrent diffusion models: hidden states are computed from clean frames during training (hidden-state prefetching) and from freshly generated clean frames during inference, both with the same frame-wise protocol. If correct, long-horizon video generation can stay globally consistent with bounded memory. The paper also reports that a simple LSTM matches or beats more recent recurrent blocks (Mamba2 and TTT) in this setting.

What carries the argument

The central mechanism is the RNN memory block inserted after attention in each DiT layer, combined with a hidden-state prefetching procedure. During training, hidden states are precomputed by running the RNN with a window size of 1 over clean frames, so all attention computations can be done in parallel; at inference the same frame-wise sliding-window protocol is used, with hidden states updated only after the final denoising step so the memory always sees clean frames. This alignment of training and inference procedures is what the paper claims removes the train/test gap.

What would settle it

A direct test would be to run a trained frame-wise RAD model autoregressively for many more steps than the training sequences, then compare the distribution of its hidden states (or a probe such as the SSIM of the first generated frame conditioned purely on memory) against hidden states computed from ground-truth frames. If the generated-state memory diverges measurably—for example, the cell-state norm or reconstruction of a held-out frame degrades with rollout length—the clean-frame prefetch equivalence is violated. A simpler version: train with noised-frame memory (the paper's own ablation)

Watch

Extended reading notes

Core claim

The paper's central claim is that a diffusion transformer augmented with an RNN memory block can generate arbitrarily long videos if the autoregressive rollout is frame-wise rather than chunk-wise, with full attention over overlapping sliding windows. The discovery is twofold: (1) with chunk-wise autoregression, where hidden states alone must bridge non-overlapping windows, LSTM's explicit separation of short-term output and long-term cell state makes it the strongest recurrent choice, outperforming Mamba2 and TTT; (2) in frame-wise mode, all three recurrent blocks perform comparably because attention carries local pixel information, letting hidden states specialize in global memory. The hea

Load-bearing premise

The load-bearing premise is that hidden states computed from ground-truth clean frames during training are representative of the hidden states the model will see from its own generated frames at inference; if the model's outputs drift, the memory becomes corrupted and the claimed 'no training and inference gap' fails.

Editorial extensions

If this is right

  • Long-video world models no longer need to choose between bounded memory and long-range consistency: a fixed-size LSTM hidden state can retain history beyond the attention window while attention preserves local detail.
  • Recurrent memory in diffusion transformers can be trained as efficiently as standard DiT, because hidden-state prefetching restores parallelism in attention computation.
  • A simple LSTM is a viable default memory block for autoregressive video diffusion, matching or exceeding more recent recurrent architectures like Mamba2 and TTT.
  • Frame-wise autoregression with overlapping windows is more important for generation quality than the choice of recurrent architecture, since local attention relieves the memory of carrying pixel-level detail.
  • The no-training-inference-gap design, with clean frames feeding memory at both stages, makes long-horizon generation stable enough to evaluate on challenging benchmarks like Memory Maze and Minecraft.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension of the paper's clean-frame memory design is to test robustness to distribution drift by training with the model's own denoised frames instead of ground-truth frames, directly probing the assumed equivalence that the prefetch relies on.
  • Because frame-wise attention erases architectural differences among RNNs, a cost-conscious follow-up could replace the LSTM with a much cheaper linear recurrence (or a learned coordinate) once local continuity is guaranteed by attention.
  • The chunk-wise result, where LSTM thrives only when memory must carry pixel detail, suggests that recurrent architectures with separate channels for local and global state are preferable whenever overlap-free windows are unavoidable, which could inform non-video sequence-modeling tasks.
  • The prefetch mechanism decouples memory updates from diffusion-noise timesteps, so memory could in principle be updated asynchronously or at lower frequency during inference, a practical optimization the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes RAD, a recurrent diffusion transformer for long video generation. RAD augments a DiT-based video diffusion model with recurrent memory blocks (LSTM, Mamba2, TTT), compares chunk-wise and frame-wise autoregressive training paradigms, and introduces a hidden-state pre-fetching scheme that allows attention computation to be parallelized during training while retaining recurrent memory. Experiments are reported on Memory Maze and Minecraft datasets, with the main empirical claims that (i) LSTM is competitive with or better than Mamba2 and TTT in this setting, (ii) frame-wise autoregression with overlapping windows substantially improves over chunk-wise autoregression, and (iii) the proposed training procedure eliminates the training/inference gap typical of recurrent diffusion models.

Significance. If the results hold, the paper makes several useful contributions to video world modeling: a controlled comparison of RNN memory blocks inside a common DiT backbone, an analysis of chunk-wise versus frame-wise autoregressive memory update, and a concrete parallelization strategy for RNN-augmented attention training. The hyperparameter table (Table 5), the efficiency analysis (Table 3), and the ablations on action conditioning (Table 4) and memory-frame noise (§6.2) are helpful and contribute to reproducibility. However, the central headline claim of 'no training and inference gap' is not established by the reported experiments, and the quantitative comparisons lack error bars or multiple seeds, so the significance of the main empirical differences cannot yet be assessed.

major comments (3)
  1. [§4.2, §4.3, Abstract] The claim of 'no training and inference gap' is unsupported. In training (§4.3), hidden states are pre-fetched from ground-truth clean frames with window size 1; in inference (§4.2), hidden states are updated from the model's own generated frames at the final DDIM step. The text in §4.2 states that this 'ensur[es] that all memory inputs consist of clean frames', but this equates 'clean' in the sense of 'denoised' with 'ground-truth'. Generated frames carry sampling error and can drift on long horizons, so the recurrent memory distribution at inference will differ from that seen during training. The assertion in §4.3 that 'the pre-fetched hidden states are equivalent to the standard DiT with RNN by sliding-window' is an assumption, not a theorem or a measurement. The ablation in §6.2 compares noised versus clean memory frames during training, which is a different manipulation and does not
  2. [Tables 1 and 2] All quantitative comparisons are reported as single runs without error bars, seeds, or significance tests. Several of the conclusions rest on small differences: in Table 2, the frame-wise PSNR values are 16.70 (Mamba2), 16.72 (TTT), and 16.59 (LSTM); in Table 1, frame-wise PSNR values are 15.35, 15.50, and 15.30. The paper claims that frame-wise variants perform similarly and that LSTM is superior in the chunk-wise setting, but without variance estimates these claims cannot be evaluated. Please report results across at least three seeds with mean and standard deviation, or provide statistical tests, for both datasets and all key configurations.
  3. [§4.3] The hidden-state pre-fetching mechanism is a central component of the training procedure, but its correctness rests on an unverified assumption. The text reads: 'Assuming the previous hidden state stacks sufficient context information, the pre-fetched hidden states are equivalent to the standard DiT with RNN by sliding-window.' This assumption is load-bearing: if pre-fetched states from clean ground-truth frames are not representative of the states the model would produce when processing its own outputs, then the training objective optimizes a different distribution from the inference procedure, and the claimed parallel-attention efficiency gain comes at the cost of a train/inference mismatch. The paper does not report any diagnostic that checks this equivalence, such as comparing pre-fetched hidden states with online-computed hidden states on the same clean training frames, or measuring
minor comments (6)
  1. [§4.3] Typo: 'bottlenece' should be 'bottleneck'.
  2. [§3.1] The LSTM equations use y_{t-1} and y_t without clearly defining the relation between y_t and the hidden state h_t. This is potentially confusing because the general RNN form at the top defines h_t and y_t separately, but the LSTM update seems to set y_t as the output. Please clarify the notation, especially since the paper later refers to 'hidden state h_t' and 'memory' interchangeably.
  3. [§6.2] The text refers to 'Table 7b', but there is no Table 7 in the main text. The ablation results in §6.2 should be presented in a properly numbered table or figure with a clear caption.
  4. [§5.1] The evaluation protocol is underspecified. For both datasets, please state whether the reported PSNR/SSIM/LPIPS are averaged over all predicted frames or over a fixed horizon, how the ground-truth comparison is aligned with the generated frames, and how many evaluation videos are used in each reported number.
  5. [§4.2, Table 5] The terms 'chunk-wise' and 'frame-wise' are defined informally in the text and Figure 3, but the exact chunk size and overlap are not given in Table 5. In particular, the attention window size is listed as 20 for both training and inference, but the chunk size used in chunk-wise experiments is not reported. Please add the chunk size and the sliding step to the hyperparameter table.
  6. [Figure 1] The right panel labels 'Hidden state pre-fetching' and 'Diffusion forward' are difficult to read and the diagram is cluttered. Consider separating the two stages into distinct subfigures and enlarging the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the load-bearing claims are empirical, and the train/inference-gap concern is an unvalidated assumption, not a circular reduction.

full rationale

This is an empirical systems paper. Its central claims — that LSTM is competitive within the RAD backbone, that frame-wise overlapping-window autoregression improves consistency over chunk-wise processing, and that hidden-state prefetching restores training parallelism — are supported by controlled experiments on Memory Maze and Minecraft, not by a derivation that reduces to its own inputs. The nearest candidate is the §4.3 assertion that, 'Assuming the previous hidden state stacks sufficient context information, the pre-fetched hidden states are equivalent to the standard DiT with RNN by sliding-window.' That sentence is an unproven modeling assumption, and the related 'no training and inference gap' claim is an empirical claim about distribution shift: training prefetch uses ground-truth clean frames, while inference updates memory from model-generated final-step frames. This is a soundness/validation gap, not an equation-level circularity: no fitted parameter is renamed as a prediction, and the claim is not forced by a mathematical identity. The only self-citation with overlapping authors, [12] (Chen et al. 2025), is used solely to specify the Minecraft data-generation protocol and does not carry the paper's load-bearing argument. No uniqueness theorem, ansatz, or benchmark result is imported from the authors' prior work in a way that forces the conclusions. The paper is therefore self-contained with respect to circularity; its weaknesses concern omitted validation of the train/inference distribution shift, not circular reasoning.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical entities, forces, or conserved quantities are introduced. The 'global memory' and 'hidden states' are standard RNN internal states; no independent falsifiable prediction outside the paper is offered for any new entity.

free parameters (4)
  • attention_window_size = 20
    Chosen by hand for Maze and Minecraft; affects the locus of local attention and the definition of chunk-wise vs frame-wise (Appendix A.1).
  • number_of_random_subsequences_N = unspecified
    In §A.3, 'we randomly sample N subsequences from the full sequence for evaluating the diffusion loss'; N is never given, yet it controls training compute and which frames receive loss.
  • DiT_depth_N = unspecified
    Fig. 2 shows '×N' layers but neither the main text nor Appendix specifies N; the architecture's memory capacity depends on it.
  • RNN_hidden_size = 1024 (LSTM)
    Appendix Table 5 lists LSTM hidden size 1024, Mamba state dim 64, TTT heads 16; these are hand-chosen capacity knobs.
assumptions (4)
  • domain assumption Pretrained VAE encodings faithfully preserve the information needed for video prediction
    The method operates in latent space using frozen Stable Diffusion/Oasis VAEs (Appendix A.4); if the VAE discards temporal/spatial detail relevant to memory, the model cannot recover it.
  • ad hoc to paper Hidden states pre-fetched from clean frames with window size 1 capture the same context as sliding-window attention with window size 20
    Stated in §4.3: 'Assuming the previous hidden state stacks sufficient context information, the pre-fetched hidden states are equivalent...' This is load-bearing for the training-parallelism trick and is never verified.
  • domain assumption Diffusion Forcing training objective (random per-frame noise levels) is a valid surrogate for autoregressive frame prediction
    The loss in §3.2 follows Chen et al. [10]; RAD inherits this without modification, relying on it to bridge partial denoising.
  • standard math Markovian Gaussian forward process for diffusion (Eq. 1) is valid
    Standard DDPM/DiT assumption, used to derive the denoising objective.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Recurrent Autoregressive Diffusion: Global Memory Meets Local Attention." pith.science (2026). https://pith.science/paper/5PKMDXMB

@misc{pith2026251112940,
  author       = {Pith},
  title        = {Pith review of: Recurrent Autoregressive Diffusion: Global Memory Meets Local Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5PKMDXMB}},
  note         = {Machine review of arXiv:2511.12940}
}
read the original abstract

Recent advancements in video generation has shifted from bidirectional models for short videos to autoregressive ones for ultra long video generation. Previous models, which usually use sliding window attention to restrict inference cost, lack effective memory compression and retrieval for long-term generation beyond the window size, leading to issues of forgetting and spatiotemporal inconsistencies. To enhance the retention of historical information with a fixed memory budget, we additionally incorporate temporal recurrent neural network (RNN) layers into the diffusion transformer (DiT) model. Specifically, we found that a LSTM layer after attention at each DiT layer achieves comparable performance to other state-of-the-art RNN blocks, such as Test-Time Training (TTT) and Mamba2. Moreover, existing diffusion-RNN approaches often suffer from performance degradation due to training-inference gap or the lack of overlap across windows. To address these limitations, we propose a novel Recurrent Autoregressive Diffusion (RAD) framework, which leverages recurrent blocks for memory update and retrieval and preserves local details by full attention on overlapping sliding windows, with no training and inference gap. Experiments on Memory Maze and Minecraft datasets demonstrate the superiority for long video generation by our framework with global memory and local attention.

Figures

Figures reproduced from arXiv: 2511.12940 by the authors.

Figure 1
Figure 1. Training paradigm for Recurrent Autoregressive Diffusion with global memory and local attention: The model has three components in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Recurrent Autoregressive Diffusion model architecture [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of effective temporal attention maps for [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Top view of Memory Maze data Maze Dataset For our small-scale experimental setting, we employ the Memory Maze dataset [32], which consists of approximately 30000 training videos, each depicting agent navigation within a 15×15 maze environment and comprising 1000 frames…
Figure 5
Figure 5. Figure 5: Visualization results on Minecraft dataset [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparison of chunk-wise TTT and frame-wise TTT [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Adding persistently updated, supervised world-state register tokens to streaming multi-agent diffusion improves cross-agent consistency and visual quality in two-agent Minecraft generation.

  2. AlayaWorld: Interactive Long-Horizon World Modeling - Full Technical Report (v1.1)

    cs.AI 2026-08 conditional novelty 5.0 of 10

    AlayaWorld v1.1 replaces depth-warped spatial memory with a streaming 3D point cache and aligns all conditioning signals to the causal VAE latent space, reporting the best WBench consistency score of 89.5.

  3. From Pixels to States: Rethinking Interactive World Models as Game Engines

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Interactive world models are reorganized around the game-engine action-state-observation loop, and a 90-hour Black Myth: Wukong dataset with frame-aligned actions, ground-truth states, and observations is introduced.

  4. Reinforcement Learning: From Algorithms To Foundation Models

    cs.AI 2026-07 conditional novelty 3.0 of 10

    A dissertation uniting the author's published results: non-exploitable Nash-DQN policies and the FightLadder benchmark for games, plus diffusion/consistency-model world models for RL — a compilation rather than new results.

Reference graph

Works this paper leans on

56 extracted references · 19 linked inside Pith · cited by 4 Pith papers

  1. [1]

    V -jepa 2: Self-supervised video models enable understanding, prediction and planning,

    Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba, Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, V asil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Y ong Li, X...

  2. [2]

    Cosmos-reason1: From physical common sense to embodied reasoning.arXiv preprint arXiv:2503.15558, 2025

    Alisson Azzolini, Hannah Brandon, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, Francesco Ferroni, Rama Govindaraju, et al. Cosmos-reason1: From physical common sense to embodied reasoning.arXiv preprint arXiv:2503.15558, 2025. 2

    arXiv 2025

  3. [3]

    Philip J. Ball, Jakob Bauer, Frank Belletti, Bethanie Brownfield, Ariel Ephrat, Shlomi Fruchter, Agrim Gupta, Kristian Holsheimer, Aleksander Holynski, Jiri Hron, Christos Kaplanis, Marjorie Limont, Matt McGill, Y anko Oliveira, Jack Parker-Holder, Frank Perbet, Guy Scully, Jeremy Shar, Stephen Spencer, Omer Tov, Ruben Villegas, Emma Wang, Jessica Y ung, ...

    2025

  4. [4]

    Navigation world models.arXiv preprint arXiv:2412.03572, 2024

    Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Y ann LeCun. Navigation world models.arXiv preprint arXiv:2412.03572, 2024. 2

    arXiv 2024

  5. [5]

    Revisiting feature prediction for learning visual representations from video, 2024

    Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Y ann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video, 2024. 1

    2024

  6. [6]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Y am Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023. 2, 3

    arXiv 2023

  7. [7]

    Align your latents: High-resolution video synthesis with latent diffusion mod- els

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion mod- els. InProceedings of the IEEE/CVF Conference on Computer V ision and P attern Recognition, pages 22563–22575, 2023. 2

    2023

  8. [8]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Y ufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. 1, 2

    2024

Show all 56 references
  1. [9]

    Gamegen-x: Interactive open-world game video generation.arXiv preprint arXiv:2411.00769, 2024

    Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamegen-x: Interactive open-world game video generation.arXiv preprint arXiv:2411.00769, 2024. 2

    2024 arXiv

  2. [10]

    Diffusion forcing: Next-token prediction meets full-sequence diffusion.Advances in Neural Information Processing Systems, 37:24081–24125,

    Boyuan Chen, Diego Mart´ı Mons´o, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion.Advances in Neural Information Processing Systems, 37:24081–24125,

  3. [11]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models

    Haoxin Chen, Y ong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer V ision and P attern Recognition, pages 7310–73...

    2024

  4. [12]

    Learning world models for interactive video generation, 2025

    Taiye Chen, Xun Hu, Zihan Ding, and Chi Jin. Learning world models for interactive video generation, 2025. 2, 6

    2025

  5. [13]

    Self-forcing++: Towards minute-scale high-quality video generation.arXiv preprint arXiv:2510.02283, 2025

    Justin Cui, Jie Wu, Ming Li, Tao Y ang, Xiaojie Li, Rui Wang, Andrew Bai, Y uanhao Ban, and Cho-Jui Hsieh. Self-forcing++: Towards minute-scale high-quality video generation.arXiv preprint arXiv:2510.02283, 2025. 2

    2025 arXiv

  6. [14]

    One-minute video generation with test-time training

    Karan Dalal, Daniel Koceja, Jiarui Xu, Y ue Zhao, Shihao Han, Ka Chun Cheung, Jan Kautz, Y ejin Choi, Y u Sun, and Xiaolong Wang. One-minute video generation with test-time training. In 2025 IEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR), pages 17702–1...

    2025

  7. [15]

    Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. InInternational Conference on Machine Learning (ICML), 2024. 1

    2024

  8. [16]

    Oasis: A universe in a transformer

    Decart, Etched, Julian Quevedo, Quinn McIntyre, Spruce Campbell, Xinlei Chen, and Robert Wachen. Oasis: A universe in a transformer. 2024. 1, 2, 4

    2024

  9. [17]

    Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752,

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752,

    arXiv

  10. [18]

    Mineworld: a real-time and open-source interactive world model on minecraft.arXiv preprint arXiv:2504.08388, 2025

    Junliang Guo, Y ang Y e, Tianyu He, Haoyu Wu, Y ushu Jiang, Tim Pearce, and Jiang Bian. Mineworld: a real-time and open-source interactive world model on minecraft.arXiv preprint arXiv:2504.08388, 2025. 2

    2025 arXiv

  11. [19]

    Minerl: A large-scale dataset of minecraft demonstrations

    William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela V eloso, and Ruslan Salakhutdi- nov. Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv:1907.13440, 2019. 6

    1907 arXiv

  12. [20]

    Recurrent world models facilitate policy evolution.Advances in neural information processing systems, 31, 2018

    David Ha and J¨urgen Schmidhuber. Recurrent world models facilitate policy evolution.Advances in neural information processing systems, 31, 2018. 2

    2018

  13. [21]

    Mastering atari with discrete world models.arXiv preprint arXiv:2010.02193, 2020

    Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models.arXiv preprint arXiv:2010.02193, 2020. 2

    2010 arXiv

  14. [22]

    Video diffusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. InAdvances in Neural Information Processing Systems, pages 8633–8646. Curran Associates, Inc., 2022. 2

    2022

  15. [23]

    Long short-term memory.Neural Computation, 9(8):1735–1780, 1997

    Sepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory.Neural Computation, 9(8):1735–1780, 1997. 2

    1997

  16. [24]

    Cogvideo: Large-scale pretraining for text-to-video generation with transformers.arXiv preprint arXiv:2205.15868, 2022

    Y u Hong, Jing Wei, Xing Liu, Xiaodi Wang, Y utong Bai, Haitao Li, Ming Zhang, and Hao Xu. Cogvideo: Large-scale pretraining for text-to-video generation with transformers.arXiv preprint arXiv:2205.15868, 2022. 2

    2022 arXiv

  17. [25]

    Gaia-1: A generative world model for autonomous driving

    Anthony Hu, Lloyd Russell, Hudson Y eo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080, 2023. 2

    2023 arXiv

  18. [26]

    Self forcing: Bridging the train-test gap in autoregressive video diffusion, 2025

    Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion, 2025. 2

    2025

  19. [27]

    How far is video generation from world model: A physical law perspective.arXiv preprint arXiv:2411.02385, 2024

    Bingyi Kang, Y ang Y ue, Rui Lu, Zhijie Lin, Y ang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How far is video generation from world model: A physical law perspective.arXiv preprint arXiv:2411.02385, 2024. 2

    2024 arXiv

  20. [28]

    Auto-encoding variational bayes, 2022

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. 2

    2022

  21. [29]

    Genie envisioner: A unified world foundation platform for robotic manipulation.arXiv preprint arXiv:2508.05635, 2025

    Y ue Liao, Pengfei Zhou, Siyuan Huang, Donglin Y ang, Sheng- cong Chen, Y uxin Jiang, Y ue Hu, Jingbin Cai, Si Liu, Jianlan Luo, Liliang Chen, Shuicheng Y an, Maoqing Y ao, and Guanghui Ren. Genie envisioner: A unified world foundation platform for robotic manipulation.arXiv p...

    2025 arXiv

  22. [30]

    Cosmos world foundation model platform for physical ai, 2025

    NVIDIA, :, Niket Agarwal, Arslan Ali, Maciej Bala, Y ogesh Bal- aji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Y ongxin Chen, Yin Cui, Yifan Ding, Daniel Dworakowski, Jiaojiao Fan, Michele Fenzi, Francesco Ferroni, Sanja Fidler, Dieter Fox, Songwei Ge, Y unhao Ge, Ji...

    2025

  23. [31]

    Genie 2: A large-scale foundation world model

    Jack Parker-Holder, Philip Ball, Jake Bruce, Vibhavari Dasagi, Kristian Holsheimer, Christos Kaplanis, Alexandre Moufarek, Guy Scully, Jeremy Shar, Jimmy Shi, Stephen Spencer, Jessica Y ung, Michael Dennis, Sultan Kenjeyev, Shangbang Long, Vlad Mnih, Harris Chan, Maxime Gazeau...

    2024

  24. [32]

    Evaluating long-term memory in 3d mazes.arXiv preprint arXiv:2210.13383, 2022

    Jurgis Pasukonis, Timothy Lillicrap, and Danijar Hafner. Evaluating long-term memory in 3d mazes.arXiv preprint arXiv:2210.13383, 2022. 5

    2022 arXiv

  25. [33]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023. 1, 2, 4

    2023

  26. [34]

    Open-sora 2.0: Training a commercial-level video genera- tion model in $200k.arXiv preprint arXiv:2503.09642, 2025

    Xiangyu Peng, Zangwei Zheng, Chenhui Shen, Tom Y oung, Xinying Guo, Binluo Wang, Hang Xu, Hongxin Liu, Mingyan Jiang, Wenjun Li, Y uhui Wang, Anbang Y e, Gang Ren, Qianran Ma, Wanying Liang, Xiang Lian, Xiwen Wu, Y uting Zhong, Zhuangyan Li, Chaoyu Gong, Guojun Lei, Leijun Che...

    2025 arXiv

  27. [35]

    Long-context state-space video world models, 2025

    Ryan Po, Y otam Nitzan, Richard Zhang, Berlin Chen, Tri Dao, Eli Shechtman, Gordon Wetzstein, and Xun Huang. Long-context state-space video world models, 2025. 1, 2, 5, 9

    2025

  28. [36]

    Hierarchical text-conditional image generation with clip latents, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022. 2

    2022

  29. [37]

    Cosmos-drive-dreams: Scalable synthetic driving data generation with world foundation models, 2025

    Xuanchi Ren, Yifan Lu, Tianshi Cao, Ruiyuan Gao, Shengyu Huang, Amirmojtaba Sabour, Tianchang Shen, Tobias Pfaff, Jay Zhangjie Wu, Runjian Chen, Seung Wook Kim, Jun Gao, Laura Leal-Taixe, Mike Chen, Sanja Fidler, and Huan Ling. Cosmos-drive-dreams: Scalable synthetic driving d...

    2025

  30. [38]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR), pages 10684–10695, 2022. 2

    2022

  31. [39]

    U-net: Convolutional networks for biomedical image segmentation,

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation,

  32. [40]

    History-guided video diffusion,

    Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion,

  33. [41]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Y u Lu, Shengfeng Pan, Wen Bo, and Y unfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024. 4

    2024

  34. [42]

    Learning to (learn at test time): Rnns with expressive hidden states, 2025

    Y u Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Geng- han Zhang, Y ann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi 10 Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (learn at test time): Rnns with expressive hidden states, 2025. 1

    2025

  35. [43]

    Generating bigger and better worlds

    World Labs team. Generating bigger and better worlds. 2025. 1

    2025

  36. [44]

    Diffusion models are real-time game engines.arXiv preprint arXiv:2408.14837, 2024

    Dani V alevski, Y aniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines.arXiv preprint arXiv:2408.14837, 2024. 2

    2024 arXiv

  37. [45]

    Lingen: Towards high-resolution minute-length text-to-video generation with linear computational complexity

    Hongjie Wang, Chih-Y ao Ma, Y en-Cheng Liu, Ji Hou, Tao Xu, Jialiang Wang, Felix Juefei-Xu, Y aqiao Luo, Peizhao Zhang, Tingbo Hou, et al. Lingen: Towards high-resolution minute-length text-to-video generation with linear computational complexity. InProceedings of the Computer...

    2025

  38. [46]

    Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13 (4):600–612, 2004

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13 (4):600–612, 2004. 6

    2004

  39. [47]

    Embed to control: A locally linear latent dynamics model for control from raw images.Advances in neural information processing systems, 28, 2015

    Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images.Advances in neural information processing systems, 28, 2015. 2

    2015

  40. [48]

    ivideogpt: Interactive videogpts are scalable world models.Advances in Neural Information Processing Systems, 37:68082–68119, 2024

    Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. ivideogpt: Interactive videogpts are scalable world models.Advances in Neural Information Processing Systems, 37:68082–68119, 2024. 2

    2024

  41. [49]

    Worldmem: Long-term consistent world simulation with memory, 2025

    Zeqi Xiao, Y ushi Lan, Yifan Zhou, Wenqi Ouyang, Shuai Y ang, Y anhong Zeng, and Xingang Pan. Worldmem: Long-term consistent world simulation with memory, 2025. 2

    2025

  42. [50]

    Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Y ang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Y uanming Y ang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024. 2

    2024 arXiv

  43. [51]

    Freeman, Fredo Durand, Eli Shechtman, and Xun Huang

    Tianwei Yin, Qiang Zhang, Richard Zhang, William T. Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer V ision and P attern Recognition (CVPR), pages ...

    2025

  44. [52]

    Context as memory: Scene-consistent interactive long video generation with memory retrieval.arXiv preprint arXiv:2506.03141, 2025

    Jiwen Y u, Jianhong Bai, Yiran Qin, Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Context as memory: Scene-consistent interactive long video generation with memory retrieval.arXiv preprint arXiv:2506.03141, 2025. 2

    2025 arXiv

  45. [53]

    Gamefactory: Creating new games with generative interactive videos.arXiv preprint arXiv:2501.08325, 2025

    Jiwen Y u, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Gamefactory: Creating new games with generative interactive videos.arXiv preprint arXiv:2501.08325, 2025. 2

    2025

  46. [54]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018. 6

    2018

  47. [55]

    Freeman, and Hao Tan

    Tianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang, Fujun Luan, Songlin Y ang, Kalyan Sunkavalli, William T. Freeman, and Hao Tan. Test-time training done right, 2025. 1, 2, 5

    2025

  48. [56]

    Open-sora: Democratizing efficient video production for all.arXiv preprint arXiv:2412.20404, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Y ang, Chenhui Shen, Shenggui Li, Hongxin Liu, Y ukun Zhou, Tianyi Li, and Y ang Y ou. Open-sora: Democratizing efficient video production for all.arXiv preprint arXiv:2412.20404, 2024. 2, 4 11 Recurrent Autoregressive Diffusion: Global Memo...

    2024 arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.