Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

Long-Context State-Space Video World Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a block-wise state-space scan plus local causal attention lets a video world model recall scenes from hundreds of frames earlier at constant per-frame cost.

desk verdict A genuinely useful architecture for long-context video world models, with honest but under-emphasized limits on memory beyond the training context. read the letter →

arxiv 2505.20171 v1 pith:UXRSKYHI submitted 2025-05-26 cs.CV

Ryan Po , Yotam Nitzan , Richard Zhang , Berlin Chen , Tri Dao , Eli Shechtman , Gordon Wetzstein , Xun Huang This is my paper · ORCID
classification cs.CV
keywords videoworldmodelsstate-spacediffusionforcinglong-termmemoryautoregressivegenerationspatialretrievalMazeMamba
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that an autoregressive video world model can keep long-term memory without paying attention's quadratic price, by replacing global temporal attention with a block-wise state-space model (SSM) scan while retaining dense local causal attention between neighboring frames. The architecture keeps per-frame inference time and memory constant as the generated video grows, and training cost grows only linearly with sequence length. On the Memory Maze retrieval and reasoning benchmarks, the model recovers previously visited regions with quality close to a full-context causal transformer (retrieval PSNR 30.8 versus 32.6), and on Minecraft reasoning it beats the diffusion-forcing transformer baseline. If correct, this removes the main scaling obstacle to interactive world models that simulate persistent environments.

What carries the argument

The load-bearing object is the block-wise SSM scan. The flattened token sequence of each video is partitioned into blocks of size $(b_h, b_w, T)$, and each block is scanned independently by a selective state-space model, so temporally adjacent tokens are only a block apart in scanning order instead of a full frame apart, and each spatial block owns a separate state that increases the total memory capacity. This is paired with frame-local causal attention that lets tokens in one frame attend to the same frame and the previous $k$ frames, compensating for the well-known weak associative recall of SSMs and keeping adjacent frames coherent. A modified diffusion-forcing training scheme, where a random-length prefix of frames is kept completely clean and loss is computed only on noised frames, forces the model to read distant clean context. Together these mechanisms make training linear in sequence length and inference constant per frame.

What would settle it

Run the same retrieval protocol on a maze with much denser visual detail (more objects, textures, higher resolution) and check whether PSNR stays flat as retrieval distance grows to 800 frames; a clear drop with scene complexity or horizon would show the fixed block state is the bottleneck. A complementary probe is to decode the SSM state and test whether object positions can be linearly read out from it.

Watch

Extended reading notes

Core claim

The paper's central claim is that the temporal memory of a video world model should be carried by causal state-space models, not by attention, provided the scan order is restructured so that memory and spatial coherence are traded off deliberately. A block-wise scan partitions the flattened spatio-temporal token sequence into spatial blocks of size $(b_h, b_w, T)$ and runs an independent SSM scan along time within each block; temporally adjacent tokens become close in scanning order, and each block gets its own fixed-size state, which enlarges total state capacity. A frame-local causal attention module (a window of the previous $k=10$ frames) then restores short-range coherence and per-frame visual quality. The authors also modify diffusion forcing so that a random-length prefix of frames is kept completely clean during training, forcing the model to learn to rely on distant clean context rather than only nearby noisy frames. On Memory Maze and TECO Minecraft, this combination yields long-horizon spatial retrieval and reasoning that approaches full-context causal transformers; on a long-horizon FVD evaluation the paper reports 38.9, below the 45.1 of the full-context causal transformer.

Load-bearing premise

The load-bearing premise is that a fixed-size compressed state per spatial block can store exact scene layout, wall geometry, and object positions well enough for hundreds of frames, so that revisits are reproduced rather than guessed.

Editorial extensions

If this is right

  • A world model built this way can reproduce an area the agent saw hundreds of frames earlier, as the retrieval and reasoning experiments show.
  • Per-frame inference time and memory stay constant as rollout length grows, so indefinite interactive generation no longer gets slower over time.
  • Training on longer context costs only linearly, making longer memory affordable instead of quadratically prohibitive.
  • The remaining gap to a full-context causal transformer is small (about 1.8 dB PSNR on retrieval and 0.6 dB on reasoning), and on FVD the hybrid model is reported ahead.
  • A practical consequence is that game-like environments can keep a persistent layout when the camera turns away and later returns, rather than regenerating inconsistent content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: the block-wise scan is effectively a way to enlarge the SSM's fixed state without changing per-frame cost, so the memory ceiling should scale with the number of blocks and the information density each block must compress; a natural next test is whether the gains hold in high-resolution, object-dense scenes.
  • A testable extension: vary the random clean-prefix length in training and measure retrieval accuracy at different retrieval distances; the paper's design implies the prefix distribution sets the memory horizon the model learns.
  • An implication the paper leaves implicit is that the same hybrid scan could be applied to other causal sequence generators beyond video diffusion, such as token-based autoregressive world models, wherever long-range spatial consistency matters.
  • Because the retrieval evaluation uses exactly invertible action sequences, an open question is how memory behaves under partial reversals or noisy actions; the Minecraft reasoning experiment only partially addresses this.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents an autoregressive video diffusion world model that replaces full-sequence attention with a block-wise causal state-space model (SSM) scan plus local frame attention, and introduces a training scheme that keeps a random clean prefix during diffusion forcing. The authors claim constant per-frame inference cost and linear training cost while retaining long-term memory, and they evaluate spatial retrieval and reasoning on Memory Maze and Minecraft, reporting that the proposed method outperforms sub-quadratic baselines and approaches a full-context causal transformer.

Significance. The architecture addresses a real bottleneck: video world models based on causal attention either lose memory under sliding-window inference or incur linear per-frame costs with full context. The block-wise scan is a sensible way to increase SSM state capacity for spatial detail, and the clean-prefix training heuristic is a plausible mechanism for encouraging long-range dependence. The complexity analysis in Table 1 and Fig. 8 is clear, and the retrieval/reasoning tasks are well chosen to expose memory failures. The paper is also honest about its limitations (no extrapolation beyond training context, low-resolution experiments). If the empirical margins survive a proper variance analysis, the method is a useful step toward efficient long-horizon world models.

major comments (3)
  1. [Sec. 5.3, Sec. 6, Table 1] All memory evaluations are conducted within the training context length: Maze retrieval uses 400 context + 400 generated frames, Maze reasoning uses 576 + 224, Minecraft uses 100 + 50, while training uses 400/800 frames for Maze and 300 frames for Minecraft (Sec. S2). The paper's own Limitation states that 'our method cannot effectively handle memory longer than the training context length.' Consequently, the 'Long Memory ✓' in Table 1 and the abstract's claim of maintaining long-term memory at constant per-frame cost are not supported for rollouts exceeding the training context. Because a sliding-window causal transformer also has constant per-frame cost but no memory, the claimed advantage hinges on memory beyond the window, yet the experiments never test memory beyond the training context. Please either qualify the claim to 'memory up to the training context length' or add an experiment that generates past this horizon (e.g., train on 400 frames and generate 800, or apply a length-extrapolation method).
  2. [Tables 2-5, Table S1] No error bars, confidence intervals, or number of seeds are reported, despite diffusion sampling being stochastic. The margins over the strongest sub-quadratic baseline are small in the reasoning task (Table 3: PSNR 28.2 vs. 27.5 for Mamba2 + Frame Local Attn; Table 4: 17.8 vs. 17.1 for DFoT). Without variance estimates, these differences cannot be distinguished from sampling noise, which weakens the central claim that the method 'surpasses baselines.' Please report mean and standard deviation over at least three seeds, or provide per-trajectory scatter plots.
  3. [Table 5 / Sec. 5.4] The ablation row 'Ours w/o block-wise scan' reports exactly the same scores as 'Mamba2 + Frame Local Attn' in Table 3 (SSIM 0.845, LPIPS 0.113, PSNR 27.5), so the entire improvement over the strongest sub-quadratic baseline is attributable to the block-wise scan, and that improvement is only 0.7 PSNR. Moreover, the block sizes (bh, bw) per layer are never specified in Sec. 4.1 or Sec. S2, so the method is not reproducible and the claimed temporal-memory versus spatial-coherence trade-off cannot be checked. Please specify the per-layer block sizes and, ideally, show sensitivity to this hyperparameter.
minor comments (5)
  1. [Fig. 6 caption] The second sentence of the caption says 'retrieval task' but the task uses 576 context frames and 224 generated frames, which is the reasoning task described in Sec. 5.2; please correct the caption.
  2. [Table 5] The table header says 'Reasoning (200 Frames)' while the corresponding results in Table 3 and Sec. 5.2 use 224 frames; please align the numbers.
  3. [Sec. 5.3] The text says 'For both tasks, we compare our method against baselines with sub-quadratic training complexity' immediately before describing the full-context causal transformer as a reference; please clarify that the reference model is not sub-quadratic.
  4. [Table 1] The row for 'Bidirectional attention' lists 'Cubic' for the AR inference complexity, but the text in Sec. 1 says bidirectional attention has quadratic complexity when generating a single frame; please reconcile the table with the text.
  5. [Sec. 5.2 / Tables 2-4] Please state the number of evaluation trajectories and the random seed(s) used for the retrieval and reasoning metrics, since the variance of these metrics is not otherwise assessable.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: long-memory capability is evaluated against ground-truth retrieval and reasoning targets, and the architecture's construction does not encode those targets.

full rationale

I walked the paper's derivation chain. Section 4 defines the block-wise SSM scan, frame local attention, and long-context training; Section 5 then evaluates the model on spatial retrieval and reasoning tasks against ground-truth frames from Memory Maze and Minecraft, with ablations in Table 5. The retrieval task reverses a held-out context trajectory, and the reasoning task continues with random actions, so neither target is fed into the model or used to fit the architecture. The claimed constant per-frame inference cost follows from the fixed-length KV-cache and fixed-dimensional SSM state by construction, but it is also empirically measured in Figure 8 and does not substitute for the memory evaluation. The only self-citation to the authors' prior work is reference [86], used as background for autoregressive video diffusion and timestep distillation; it is not load-bearing evidence for the new SSM state capacity. The Limitations statement that 'our method cannot effectively handle memory longer than the training context length' is an honest scope restriction, not a circular derivation; it qualifies the strength of the long-memory claim but does not make the measured results equivalent to the method's inputs. No load-bearing step reduces to a fitted parameter, a self-citation chain, or a definition of the target metric in terms of the model output.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical entities, conserved quantities, or forces are introduced. The architecture-specific hand-tuned choices (block sizes, window size, training schedule) are the main parameters the central claim depends on, and they are treated as free parameters here.

free parameters (4)
  • Block sizes (bh, bw) per layer
    Chosen by hand to balance temporal memory and spatial coherence; actual values are not reported and no sweep is shown (Sec 4.1).
  • Frame local attention window size k = 10
    Set to k=10 in all experiments (Supp. S2); no ablation on k is reported.
  • Long-context training prefix probability p = 0.5
    Chosen for all experiments (Supp. S2), with the additional rule that the clean prefix length must exceed half the sequence length. This is an ad hoc training choice.
  • Frame chunk size = 5
    Frames are grouped into chunks of 5 for speed, making the effective attention window 10 (Supp. S2).
assumptions (4)
  • standard math Diffusion forcing training with per-frame independent noise levels is a valid autoregressive training objective.
    The method builds directly on diffusion forcing (Ref 10) and prior art (Ref 59); if this training framework were invalid, the architecture would inherit the problem.
  • domain assumption The datasets (Memory Maze, TECO Minecraft) present static, deterministic environments where revisiting a location yields the same appearance, so frame prediction can be scored against ground truth using appearance metrics.
    The retrieval and reasoning tasks in Sec 5.2 assume the world is static and ground truth frames are unique; this is true for the chosen benchmarks but not for dynamic open worlds.
  • domain assumption The Mamba selective SSM recurrence (Eq 5) can be trained with parallel scans and has enough expressivity for visual generation when combined with local attention.
    The authors rely on Mamba's properties (Ref 20) and on prior hybrid SSM/attention results from language modeling; this is an empirical assumption not proven in the paper.
  • ad hoc to paper Keeping a random clean prefix during training encourages the model to attend to distant frames rather than relying on noisy local frames.
    Motivated in Sec 4.2 with empirical ablation support in Table 5, but no theoretical guarantee is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Long-Context State-Space Video World Models." pith.science (2026). https://pith.science/paper/UXRSKYHI

@misc{pith2026250520171,
  author       = {Pith},
  title        = {Pith review of: Long-Context State-Space Video World Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UXRSKYHI}},
  note         = {Machine review of arXiv:2505.20171}
}
read the original abstract

Video diffusion models have recently shown promise for world modeling through autoregressive frame prediction conditioned on actions. However, they struggle to maintain long-term memory due to the high computational cost associated with processing extended sequences in attention layers. To overcome this limitation, we propose a novel architecture leveraging state-space models (SSMs) to extend temporal memory without compromising computational efficiency. Unlike previous approaches that retrofit SSMs for non-causal vision tasks, our method fully exploits the inherent advantages of SSMs in causal sequence modeling. Central to our design is a block-wise SSM scanning scheme, which strategically trades off spatial consistency for extended temporal memory, combined with dense local attention to ensure coherence between consecutive frames. We evaluate the long-term memory capabilities of our model through spatial retrieval and reasoning tasks over extended horizons. Experiments on Memory Maze and Minecraft datasets demonstrate that our approach surpasses baselines in preserving long-range memory, while maintaining practical inference speeds suitable for interactive applications.

Figures

Figures reproduced from arXiv: 2505.20171 by the authors.

Figure 1
Figure 1. Failure case of existing video world models. With￾out long-term memory, previously observed regions may appear altered or inconsistent upon revisiting. of their attention mechanisms. This limitation hinders their ability to simulate a persistent world with long-term consis￾tency. For example, when using an existing video world model to simulate a game, the entire environment might completely change after a player si… view at source ↗
Figure 2
Figure 2. Long memory video generation. Our method generates sharp and consistent video predictions. Conditioned on agent actions, our method can accurately reconstruct previously visited regions of an environment, while maintaining linear training complexity and constant inference costs. In contrast, although state-of-the-art diffusion forcing transformers (DFoT [59]) can generate consistent looking videos over long horizons… view at source ↗
Figure 3
Figure 3. Model architecture. Our model features a block￾wise SSM scan that divides spatial dimensions into independent scanning blocks (bh, bw), balancing temporal memory with spa￾tial coherence. This works alongside frame local attention, which enables bidirectional processing within frames while maintaining causal relationships across the previous k frames, resulting in im￾proved per-frame visual quality and temporal consi… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Improved long-context training. a) Standard diffu￾sion forcing injects independent noise levels to all frames, b) Our method keeps a random number of initial frames completely clean (ti = 0), adds independent noise to later frames, and calculates loss only on the noise…
Figure 5
Figure 5. Figure 5: Overview of retrieval task and qualitative results. The maze layout shows the context trajectory (red) and retrieval trajectory (gray), which reverses the original path. We compare each model on a retrieval task with 400 generated frames following 400 context frames. T…
Figure 6
Figure 6. Figure 6: Overview of reasoning task and qualitative results. The maze layout illustrates the context trajectory (red) and reasoning trajectory (gray)1 , which continues the context path. We compare each model on a retrieval task with 224 generated frames conditioned on 576 cont…
Figure 7
Figure 7. Figure 7: Retrieval PSNR vs. Frame Distance. Our model maintains consistent high performance comparable to full-context transformers while significantly outperforming limited-context transformers that degrade beyond training length and linear￾complexity models that lack sufficie…
Figure 8
Figure 8. Figure 8: Training and inference performance comparisons. Evaluation of training costs (left), inference memory usage (center), and inference time (right), demonstrating how our approach maintains consistent memory and computational efficiency as frame count increases compared t…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DIM-WAM: World-Action Modeling with Diverse Historical Event Memory

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    Multi-bank similarity-merged event memory plus progress supervision raises long-horizon WAM success from 28.4% to 69.8% on RMBench and full-task success from 52.5% to 80% on real Franka tasks.

  2. End-to-End Training for Autoregressive Video Diffusion via Self-Resampling

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Resampling Forcing trains autoregressive video diffusion models on self-resampled degraded histories with a causal mask, achieving stable long-horizon generation without a teacher or discriminator.

  3. M4V: Multimodal Mamba for Efficient Text-to-Video Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    M4V shows a Mamba-based text-to-video model can roughly match attention-based PyramidFlow on VBench while cutting mixer-layer FLOPs by 45% at 768x1280.

  4. Unlocking Temporal Generalization in Hamiltonian Video Dynamics Models

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Spectral normalization of the action-force map and inference-time integrator substepping let port-Hamiltonian generative networks predict forced dissipative video dynamics at step sizes far outside training.

Reference graph

Works this paper leans on

90 extracted references · 30 canonical work pages · cited by 4 Pith papers

  1. [1]

    Diffusion for world modeling: Visual details matter in atari

    Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kan- ervisto, Amos Storkey, Tim Pearce, and Franc ¸ois Fleuret. Diffusion for world modeling: Visual details matter in atari. In NeurIPS, 2024. 1

    2024

  2. [2]

    Genesis: A universal and generative physics engine for robotics and beyond, 2024

    Genesis Authors. Genesis: A universal and generative physics engine for robotics and beyond, 2024. 1

    2024

  3. [3]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. Layer normalization. arXiv preprint arXiv:1607.06450,

    arXiv

  4. [4]

    Ti- tans: Learning to memorize at test time

    Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Ti- tans: Learning to memorize at test time. arXiv preprint arXiv:2501.00663, 2024. 3, 5

    arXiv 2024

  5. [5]

    Decimamba: Exploring the length extrapolation potential of mamba

    Assaf Ben-Kish, Itamar Zimerman, Shady Abu-Hussein, Na- dav Cohen, Amir Globerson, Lior Wolf, and Raja Giryes. Decimamba: Exploring the length extrapolation potential of mamba. 2025. 8

    2025

  6. [6]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2

    arXiv 2023

  7. [7]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  8. [8]

    Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal M

    Jake Bruce, Michael D. Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal M. P. Behbahani, Stephanie Chan, Nicolas Manfred Otto Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando d...

    arXiv 2024

Show all 90 references
  1. [9]

    Gamegen-x: Interactive open-world game video generation

    Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamegen-x: Interactive open-world game video generation. In ICLR, 2025. 1, 2

    2025

  2. [10]

    Diffusion forcing: Next-token prediction meets full-sequence diffu- sion

    Boyuan Chen, Diego Mart ´ı Mons ´o, Yilun Du, Max Sim- chowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffu- sion. In NeurIPS, 2024. 2, 3

    2024

  3. [11]

    Seine: Short-to-long video diffusion model for generative transition and prediction

    Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. In ICLR,

  4. [12]

    Recurrent environment simulators

    Silvia Chiappa, S ´ebastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. In ICLR, 2017. 2

    2017

  5. [13]

    Transformers are ssms: General- ized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: General- ized models and efficient algorithms through structured state space duality. In ICML, 2024. 3, 6, 7

    2024

  6. [14]

    Oasis: A universe in a transformer

    Julian Decart, Quinn Quevedo, Spruce McIntyre, Xinlei Campbell, Robert Chen, and Wachen. Oasis: A universe in a transformer. 2024. 1, 2

    2024

  7. [15]

    Flex attention: A programming model for generating optimized attention kernels

    Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A programming model for generating optimized attention kernels. ArXiv, abs/2412.05496, 2024. 12

    2024 arXiv

  8. [16]

    The matrix: Infinite-horizon world generation with real-time moving control

    Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, and Hongyang Zhang. The matrix: Infinite-horizon world generation with real-time moving control. arXiv preprint arXiv:2412.03568, 2024. 1, 2

    2024 arXiv

  9. [17]

    Vid-gpt: Introducing gpt-style autoregres- sive generation in video diffusion models

    Kaifeng Gao, Jiaxin Shi, Hanwang Zhang, Chunping Wang, and Jun Xiao. Vid-gpt: Introducing gpt-style autoregres- sive generation in video diffusion models. arXiv preprint arXiv:2406.10981, 2024. 2

    2024 arXiv

  10. [18]

    Ca2-vdm: Efficient autoregres- sive video diffusion model with causal generation and cache sharing

    Kaifeng Gao, Jiaxin Shi, Hanwang Zhang, Chunping Wang, Jun Xiao, and Long Chen. Ca2-vdm: Efficient autoregres- sive video diffusion model with causal generation and cache sharing. arXiv preprint arXiv:2411.16375, 2024. 1, 2, 5

    2024 arXiv

  11. [19]

    Matten: Video generation with mamba- attention

    Yu Gao, Jiancheng Huang, Xiaopeng Sun, Zequn Jie, Yujie Zhong, and Lin Ma. Matten: Video generation with mamba- attention. arXiv preprint arXiv:2405.03025, 2024. 3

    2024 arXiv

  12. [20]

    Mamba: Linear-time sequence mod- eling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence mod- eling with selective state spaces. In COLM, 2024. 1, 3, 4

    2024

  13. [21]

    Photorealistic video generation with diffusion models

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Fei-Fei Li, Irfan Essa, Lu Jiang, and Jos ´e Lezama. Photorealistic video generation with diffusion models. In ECCV, 2024. 3

    2024

  14. [22]

    Recurrent world models facilitate policy evolution

    David Ha and J ¨urgen Schmidhuber. Recurrent world models facilitate policy evolution. In NeurIPS, 2018. 1, 2

    2018

  15. [23]

    Ltx-video: Realtime video latent diffusion

    Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.00103 ,

    arXiv

  16. [24]

    Dream to control: Learning behaviors by la- tent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Moham- mad Norouzi. Dream to control: Learning behaviors by la- tent imagination. In ICLR, 2020. 1

    2020

  17. [25]

    Pre-trained video generative models as world simula- tors

    Haoran He, Yang Zhang, Liang Lin, Zhongwen Xu, and Ling Pan. Pre-trained video generative models as world simula- tors. arXiv preprint arXiv:2502.07825, 2025. 1

    2025 arXiv

  18. [26]

    Latent video diffusion models for high-fidelity long video generation

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221 ,

    arXiv

  19. [27]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. In NeurIPS, 2020. 3

    2020

  20. [28]

    Video dif- fusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. In NeurIPS, 2022. 1, 2

    2022

  21. [29]

    Long short-term memory

    Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. 2

    1997

  22. [30]

    Gaia-1: A generative world model for au- tonomous driving

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gian- luca Corrado. Gaia-1: A generative world model for au- tonomous driving. arXiv preprint arXiv:2309.17080, 2023. 1, 2

    2023 arXiv

  23. [31]

    Acdit: Interpolating autoregressive con- ditional modeling and diffusion transformer

    Jinyi Hu, Shengding Hu, Yuxuan Song, Yufei Huang, Mingxuan Wang, Hao Zhou, Zhiyuan Liu, Wei-Ying Ma, and Maosong Sun. Acdit: Interpolating autoregressive con- ditional modeling and diffusion transformer. arXiv preprint arXiv:2412.07720, 2024. 1, 2

    2024

  24. [32]

    Arbitrary style transfer in real-time with adaptive instance normalization

    Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV,

  25. [33]

    Multimodal unsupervised image-to-image translation

    Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In ECCV, 2018. 5

    2018

  26. [34]

    Scope of va- lidity of psnr in image/video quality assessment

    Quan Huynh-Thu and Mohammed Ghanbari. Scope of va- lidity of psnr in image/video quality assessment. Electronics letters, 2008. 6

    2008

  27. [35]

    Pyramidal flow matching for efficient video generative modeling

    Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. In ICLR, 2025. 1, 2

    2025

  28. [36]

    How far is video generation from world model: A physical law perspective

    Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How far is video generation from world model: A physical law perspective. arXiv preprint arXiv:2411.02385, 2024. 1

    2024 arXiv

  29. [37]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In ICML, 2020. 3

    2020

  30. [38]

    Auto-encoding varia- tional bayes

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. In ICLR, 2014. 2

    2014

  31. [39]

    Videopoet: A large language model for zero-shot video gen- eration

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jose Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vigh- nesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video gen- eration. In ICML, 2024. 2

    2024

  32. [40]

    Hunyuanvideo: A systematic framework for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. 1, 2

    2024 arXiv

  33. [41]

    Efficient spatially sparse inference for conditional gans and diffusion models

    Muyang Li, Ji Lin, Chenlin Meng, Stefano Ermon, Song Han, and Jun-Yan Zhu. Efficient spatially sparse inference for conditional gans and diffusion models. Advances in neu- ral information processing systems, 35:28858–28873, 2022. 2

    2022

  34. [42]

    Clear: Conv-like linearization revs pre-trained diffusion transform- ers up

    Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Clear: Conv-like linearization revs pre-trained diffusion transform- ers up. arXiv preprint arXiv:2412.16112, 2024. 3

    2024 arXiv

  35. [43]

    Linfusion: 1 gpu, 1 minute, 16k image.arXiv preprint arXiv:2409.02097, 2024

    Songhua Liu, Weihao Yu, Zhenxiong Tan, and Xinchao Wang. Linfusion: 1 gpu, 1 minute, 16k image.arXiv preprint arXiv:2409.02097, 2024. 3

    2024 arXiv

  36. [44]

    Step-video-t2v technical re- port: The practice, challenges, and future of video founda- tion model

    Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xi- aoniu Song, Xing Chen, et al. Step-video-t2v technical re- port: The practice, challenges, and future of video founda- tion model. arXiv preprint arXiv:2502.10248, 2025. 1

    2025 arXiv

  37. [45]

    Playable video gen- eration

    Willi Menapace, Stephane Lathuiliere, Sergey Tulyakov, Aliaksandr Siarohin, and Elisa Ricci. Playable video gen- eration. In CVPR, 2021. 2

    2021

  38. [46]

    Trans- formers are sample-efficient world models

    Vincent Micheli, Eloi Alonso, and Franc ¸ois Fleuret. Trans- formers are sample-efficient world models. In ICLR, 2023. 1

    2023

  39. [47]

    Action-conditional video prediction us- ing deep networks in atari games

    Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction us- ing deep networks in atari games. In NeurIPS, 2015. 2

    2015

  40. [48]

    Ssm meets video diffusion models: Efficient long-term video generation with structured state spaces

    Yuta Oshima, Shohei Taniguchi, Masahiro Suzuki, and Yu- taka Matsuo. Ssm meets video diffusion models: Efficient long-term video generation with structured state spaces. arXiv preprint arXiv:2403.07711, 2024. 2, 3

    2024 arXiv

  41. [49]

    Genie 2: A large-scale foundation world model

    Jack Parker-Holder, Philip Ball, Jake Bruce, Vibhavari Dasagi, Kristian Holsheimer, Christos Kaplanis, Alexandre Moufarek, Guy Scully, Jeremy Shar, Jimmy Shi, Stephen Spencer, Jessica Yung, Michael Dennis, Sultan Kenjeyev, Shangbang Long, Vlad Mnih, Harris Chan, Maxime Gazeau,...

    2024

  42. [50]

    Evaluating long-term memory in 3d mazes

    Jurgis Pasukonis, Timothy Lillicrap, and Danijar Hafner. Evaluating long-term memory in 3d mazes. arXiv preprint arXiv:2210.13383, 2022. 6

    2022 arXiv

  43. [51]

    Scalable diffusion mod- els with transformers

    William S Peebles and Saining Xie. Scalable diffusion mod- els with transformers. In ICCV, 2023. 5

    2023

  44. [52]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  45. [53]

    Hierarchical text-conditional image gener- ation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents. arXiv preprint arXiv:2204.06125, 1 (2):3, 2022. 2

    2022 arXiv

  46. [54]

    Samba: Simple hybrid state space models for efficient unlimited context language modeling

    Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. ArXiv, abs/2406.07522, 2024. 5

    2024 arXiv

  47. [55]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, 2022. 2

    2022

  48. [56]

    Linear transformers are secretly fast weight programmers

    Imanol Schlag, Kazuki Irie, and J ¨urgen Schmidhuber. Linear transformers are secretly fast weight programmers. InICML,

  49. [57]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. In ICLR, 2023. 2

    2023

  50. [58]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015. 3

    2015

  51. [59]

    History-guided video diffusion

    Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion. arXiv preprint arXiv:2502.06764, 2025. 1, 2, 3, 6, 8, 12

    2025 arXiv

  52. [60]

    Score-based generative modeling through stochastic differential equa- tions

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions. In ICLR, 2021. 3

    2021

  53. [61]

    Me playing a few minutes of ai minecraft

    stealthispost. Me playing a few minutes of ai minecraft

  54. [62]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024. 2

    2024 arXiv

  55. [63]

    Learning to (learn at test time): Rnns with expressive hidden states

    Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, et al. Learning to (learn at test time): Rnns with expressive hidden states. arXiv preprint arXiv:2407.04620, 2024. 3

    2024 arXiv

  56. [64]

    Dim: Diffusion mamba for efficient high-resolution image synthesis

    Yao Teng, Yue Wu, Han Shi, Xuefei Ning, Guohao Dai, Yu Wang, Zhenguo Li, and Xihui Liu. Dim: Diffusion mamba for efficient high-resolution image synthesis. arXiv preprint arXiv:2405.14224, 2024. 3

    2024 arXiv

  57. [65]

    Diffusion models are real-time game engines

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837, 2024. 1, 2

    2024 arXiv

  58. [66]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 2, 3

    2017

  59. [67]

    Phenaki: Variable length video generation from open domain textual descriptions

    R Villegas, H Moraldo, S Castro, M Babaeizadeh, H Zhang, J Kunze, PJ Kindermans, MT Saffar, and D Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In ICLR, 2023. 3

    2023

  60. [68]

    Lingen: Towards high-resolution minute-length text-to-video generation with linear computa- tional complexity

    Hongjie Wang, Chih-Yao Ma, Yen-Cheng Liu, Ji Hou, Tao Xu, Jialiang Wang, Felix Juefei-Xu, Yaqiao Luo, Peizhao Zhang, Tingbo Hou, et al. Lingen: Towards high-resolution minute-length text-to-video generation with linear computa- tional complexity. arXiv preprint arXiv:2412.0985...

    2024 arXiv

  61. [69]

    Worlddreamer: Towards general world models for video generation via predicting masked to- kens

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Boyuan Wang, Xinze Chen, and Jiwen Lu. Worlddreamer: Towards general world models for video generation via predicting masked to- kens. arXiv preprint arXiv:2401.09985, 2024. 1

    2024 arXiv

  62. [70]

    Loong: Generating minute-level long videos with autoregressive lan- guage models

    Yuqing Wang, Tianwei Xiong, Daquan Zhou, Zhijie Lin, Yang Zhao, Bingyi Kang, Jiashi Feng, and Xihui Liu. Loong: Generating minute-level long videos with autoregressive lan- guage models. arXiv preprint arXiv:2410.02757, 2024. 2

    2024 arXiv

  63. [71]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 2004. 6

    2004

  64. [72]

    Art-v: Auto-regressive text-to- video generation with diffusion models

    Wenming Weng, Ruoyu Feng, Yanhui Wang, Qi Dai, Chunyu Wang, Dacheng Yin, Zhiyuan Zhao, Kai Qiu, Jian- min Bao, Yuhui Yuan, et al. Art-v: Auto-regressive text-to- video generation with diffusion models. In CVPR, 2024. 1

    2024

  65. [73]

    Daydreamer: World models for physical robot learning

    Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. In CoRL, 2023. 1

    2023

  66. [74]

    Pandora: Towards general world model with natural language actions and video states

    Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, et al. Pandora: Towards general world model with natural language actions and video states. arXiv preprint arXiv:2406.09455, 2024. 2

    2024 arXiv

  67. [75]

    Sana: Efficient high-resolution image syn- thesis with linear diffusion transformers

    Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image syn- thesis with linear diffusion transformers. arXiv preprint arXiv:2410.10629, 2024. 3

    arXiv

  68. [76]

    Diffu- sion models without attention

    Jing Nathan Yan, Jiatao Gu, and Alexander M Rush. Diffu- sion models without attention. In CVPR, 2024. 3

    2024

  69. [77]

    Videogpt: Video generation using vq-vae and trans- formers

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and trans- formers. arXiv preprint arXiv:2104.10157, 2021. 2

    2021 arXiv

  70. [78]

    Wilson Yan, Danijar Hafner, Stephen James, and P. Abbeel. Temporally consistent transformers for video generation. In ICML, 2022. 6

    2022

  71. [79]

    Learn- ing interactive real-world simulators

    Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learn- ing interactive real-world simulators. arXiv preprint arXiv:2310.06114, 1(2):6, 2023. 2

    2023 arXiv

  72. [80]

    Gated linear attention trans- formers with hardware-efficient training

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention trans- formers with hardware-efficient training. arXiv preprint arXiv:2312.06635, 2023. 3

    2023 arXiv

  73. [81]

    Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464, 2024

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464, 2024. 3, 5

    2024 arXiv

  74. [82]

    Parallelizing linear transformers with the delta rule over sequence length

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. Advances in Neural Information Processing Systems, 37:115491–115522, 2025. 3, 5

    2025

  75. [83]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 2, 12

    2024 arXiv

  76. [84]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. In ICLR, 2025. 1

    2025

  77. [85]

    Longmamba: Enhancing mamba’s long-context capabilities via training-free receptive field en- largement

    Zhifan Ye, Kejing Xia, Yonggan Fu, Xin Dong, Jihoon Hong, Xiangchi Yuan, Shizhe Diao, Jan Kautz, Pavlo Molchanov, and Yingyan Celine Lin. Longmamba: Enhancing mamba’s long-context capabilities via training-free receptive field en- largement. In ICLR, 2025. 8

    2025

  78. [86]

    From slow bidirectional to fast autoregressive video diffusion mod- els

    Tianwei Yin, Qiang Zhang, Richard Zhang, William T Free- man, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion mod- els. In CVPR, 2025. 1, 2, 3, 4, 8

    2025

  79. [87]

    Gamefactory: Creating new games with gen- erative interactive videos

    Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Gamefactory: Creating new games with gen- erative interactive videos. arXiv preprint arXiv:2501.08325,

  80. [88]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 6

    2018

  81. [89]

    Extdm: Distribution extrapolation diffu- sion model for video prediction

    Zhicheng Zhang, Junyao Hu, Wentao Cheng, Danda Paudel, and Jufeng Yang. Extdm: Distribution extrapolation diffu- sion model for video prediction. In CVPR, 2024. 2

    2024

  82. [90]

    Is sora a world simulator? a comprehensive survey on general world models and beyond

    Zheng Zhu, Xiaofeng Wang, Wangbo Zhao, Chen Min, Ni- anchen Deng, Min Dou, Yuqi Wang, Botian Shi, Kai Wang, Chi Zhang, Yang You, Zhaoxiang Zhang, Dawei Zhao, Liang Xiao, Jian Zhao, Jiwen Lu, and Guan Huang. Is sora a world simulator? a comprehensive survey on general world mod...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.