Pith. sign in

REVIEW 5 major objections 7 minor 18 cited by

The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control

T0 review · 5 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 2.7B-parameter diffusion world model generates infinitely long, controllable 720p video at 8-16 FPS, including scenes never seen in training.

desk verdict A serious streaming-diffusion mechanism and a clever data-collection platform, wrapped in overclaimed evaluation—worth refereeing, not worth believing yet. read the letter →

arxiv 2412.03568 v1 pith:MM4CP7IH submitted 2024-12-04 cs.AI

Ruili Feng , Han Zhang , Zhantao Yang , Jie Xiao , Zhilei Shu , Zhiheng Liu , Andy Zheng , Yukun Huang
show 2 more authors
Yu Liu Hongyang Zhang
This is my paper · ORCID
classification cs.AI
keywords worldmodelsvideodiffusiontransformersinfinite-horizongenerationreal-timeinteractivecontrolframe-levelactionconditioningzero-shotgeneralizationgametelemetrydatastreamingdenoising
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims the first realistic world simulator that generates continuous, uncut 720p video for arbitrarily long durations while responding to frame-level control signals in real time. The system, a 2.7B-parameter diffusion transformer trained on game telemetry paired with video frames plus large unlabeled real-world footage, streams at 8 to 16 frames per second in first- or third-person views. If correct, interactive simulated environments could be produced from data rather than hand-built game engines, and game data could transfer to real-world scenes that were never collected. The load-bearing idea is that temporal attention can be confined to a sliding window, so a diffusion model can run as a streaming process instead of generating fixed-length clips.

What carries the argument

The central mechanism is the Shift-Window Denoising Process Model (Swin-DPM), a sliding-window denoising process that turns a finite diffusion generation into an indefinitely long stream. It keeps a queue of $T$ video tokens at different noise levels, denoises them for $k$ steps, dequeues the lowest-noise token into a cache, adds a fresh Gaussian-noise token at the right end, and repeats, while cached tokens stay in the attention window to preserve continuity across windows. The Interactive Module contributes causal cross-attention so each keyboard input affects only the current and next $\omega=4$ tokens, and the Stream Consistency Model compresses the denoising trajectory to reach 8-16 FPS.

What would settle it

Run a fixed, straight-line control stream for an hour and monitor whether scene content drifts or resets, for example by checking whether a distinctive landmark stays recognizable or whether a looping route returns to the same place once the window has slid. If perceptual similarity to a fixed reference decays without bound over hours, the finite-window attention assumption is falsified.

Watch

Extended reading notes

Core claim

The core claim is that one 2.7B-parameter model, The Matrix, can act as a foundational realistic world simulator: it generates endless $1280 \times 720$ video, accepts keyboard and mouse input at frame level, renders at 8 to 16 FPS, and generalizes zero-shot to environments outside its training data, such as a car driving through an office or through water. The model is built by taking a pre-trained video diffusion transformer (a video generator that denoises latent frames through transformer attention), adding an Interactive Module that translates controls into condition tokens, applying the Shift-Window Denoising Process Model (Swin-DPM) to make generation stream through a sliding denoising queue, and distilling a Stream Consistency Model (SCM) for acceleration. Training uses the Source dataset, roughly 750k labeled action-frame clips extracted from AAA games via CPU-memory telemetry plus 1.2 million unlabeled real-world clips.

Load-bearing premise

The load-bearing premise is that every temporal dependency needed for a coherent scene fits inside a sliding window of a few video tokens; if hour-long streams require memory of events farther back than the window, the infinite-horizon claim collapses.

Editorial extensions

If this is right

  • Game development could shift from hand-built engines to learned simulators, because a world's look and response would be inferred from footage and control telemetry rather than authored.
  • Long-form video generation would no longer require stitching independent clips, since Swin-DPM extends a pre-trained diffusion transformer as a single continuous stream.
  • Simulation becomes possible for settings where continuous, action-labeled data cannot be collected, such as indoor driving or emergency scenes, via zero-shot transfer from games.
  • Embodied-AI training data could take the same shape as gameplay data, since the model also responds to per-frame robot-arm joint actions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that if Swin-DPM's window holds, any pre-trained video diffusion transformer could be turned into an interactive, infinite-horizon world by fine-tuning these three modules, making long-form interactivity a fine-tuning problem rather than a new architecture.
  • The reported metrics average over 2,048 seconds of generated video, so they do not directly test hour-scale consistency; a natural extension is measuring object and layout persistence over multiple hours, where the finite window predicts gradual drift.
  • GameData's CPU-memory telemetry should transfer to other games with readable internal state, which could make frame-level action labels nearly free and widen the supply of paired training data for world models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper presents "The Matrix," a 2.7B-parameter diffusion-transformer world model trained on paired AAA game footage (Forza Horizon 5, Cyberpunk 2077) and unlabeled real-world video, with three technical contributions: an Interactive Module for frame-level keyboard control, a Shift-Window Denoising Process Model (Swin-DPM) for streaming/infinite video generation from a pre-trained DiT, and a Stream Consistency Model (SCM) distillation for real-time inference. The manuscript claims continuous 720p generation at 8-16 FPS, hour-long uncut sequences, and zero-shot generalization to unseen scenes such as a car driving indoors.

Significance. If the central claims were substantiated, this would be a notable engineering result: a relatively small open-source-style world model with real-time interactive control and domain generalization would be valuable to the community. The GameData platform and Source dataset are potentially useful contributions, and the promise to release code, data, and checkpoints is a strength. However, the paper's evidence is almost entirely qualitative or internally ablative. There are no comparisons to prior simulators, no error bars, no long-horizon coherence metrics, and no hardware configuration for the real-time claim. The 'infinite-horizon' claim in particular rests on an explicit finite-window assumption in Section 3.1 that is neither validated nor structurally capable of maintaining global scene consistency beyond roughly six seconds.

major comments (5)
  1. [Section 3.1, Figure 3b] The infinite-horizon claim is load-bearing and unsupported. Swin-DPM assumes temporal dependencies are confined to a finite window w (set to T in the text), and the implementation only re-appends the immediately preceding cached token at noise level 0 until the next token is dequeued. With 4 tokens per second and T tokens per window, the effective temporal context is about 6 seconds plus one token. There is no latent state, memory bank, or global consistency mechanism that could maintain scene layout, object identity, or route structure over an hour. The paper provides no drift analysis, object-persistence test, or long-horizon consistency metric; the longest quantitative or visual evidence is the 1-2.5 minute demos in Figure 6, and the referenced half-hour supplementary videos are not part of the written record. Absent such evidence, the Abstract's 'continuous, uncut hour-long sequences' and the repeated 'infinitely long' claims are not established.
  2. [Section 4.2, Table 2] The quantitative support for infinite-length generation is missing. Table 2 reports FVD, FID, CLIP, Move-LPIPS, and Move-PSNR over '2,048 seconds of randomly generated videos,' but it does not report these metrics as a function of generated sequence length, nor does it compare a 1-minute, 30-minute, or 60-minute generation against any prior simulator. The final real-time system's FVD on Forza Horizon 5 (1936.79) is substantially worse than the warmup-only model on the same scene (1891.67) and much worse than the unmodified backbone (1016.30), yet the text in Section 4.2 claims that visual quality 'still surpasses previous world simulators' without citing any comparison in the table or elsewhere.
  3. [Section 4.1, Table 2] The control-precision metrics are not sufficiently specified to be interpretable. Move-PSNR and Move-LPIPS are said to be computed 'between the generated videos and real videos with ground truth movements,' but the paper does not state whether the real videos share the same initial frame, scene geometry, camera trajectory, and control sequence as the generated videos, nor whether the comparison is frame-aligned. Without this protocol description, a low LPIPS value could reflect visual similarity to the scene rather than control precision. In addition, no error bars or multiple-seed statistics are reported, so differences such as 0.109 vs. 0.113 are not evaluable.
  4. [Section 4.2, Abstract] The real-time claim is not reproducible as reported. The paper gives inference speeds of 8-16 FPS (or 'up to 16 FPS' in the Abstract) without specifying the GPU hardware, the number of GPUs, the resolution and VAE decoding cost at inference, the batch size, or the measurement protocol. Table 2 mixes units (e.g., '48 frames / 34 Seconds' for the backbone versus '0.8 FPS' for Swin-DPM), and no timing breakdown for the Swin-DPM window update and SCM distillation is provided. The real-time capability is a central contribution and needs a precise reproducibility statement.
  5. [Section 4.3, Figure 7] The zero-shot generalization claim is demonstrated only with still frames. The examples of driving indoors or controlling a person in an unseen scene are not accompanied by any quantitative measure of control accuracy, temporal stability, or scene consistency, and there is no comparison to a non-generalizing baseline. As stated, Figure 7 shows qualitative outputs but does not substantiate 'strong domain generalization' or the BMW-through-an-office claim in the Abstract.
minor comments (7)
  1. [Section 4.2 heading] The heading 'Infinete-Horizon World Generation' contains a typo; it should be 'Infinite-Horizon.'
  2. [Author affiliation] The affiliation line contains 'Vector Insititute'; this should be corrected to 'Vector Institute.'
  3. [Appendix B.1 heading] The heading 'Controling' should be 'Controlling.'
  4. [Appendix B.1] The text refers to 'dessert' in the list of Forza Horizon 5 scenes; the intended word is 'desert.'
  5. [Section 3.2 vs. Appendix B.2.1] The dataset size is inconsistent: Section 3.2 states 750k labeled and 1.2M unlabeled samples, while Appendix B.2.1 reports about 1.2M Forza Horizon 5 pairs and about 1M Cyberpunk 2077 pairs. The relationship between these numbers (e.g., before and after filtering) should be stated explicitly.
  6. [Section 4.2] The sentence 'with additional half-hour examples available in Supplementary Videos' is not verifiable from the manuscript; the supplementary material only mentions compressed demos. Either include a link or describe the content and length of each supplementary video.
  7. [Table 1] Table 1 assigns 'Control Generalization: Yes' only to The Matrix, but the criterion for this entry is not defined, and the comparisons in the table are self-assessed rather than measured. Please define each feature precisely and, where possible, cite the original papers' reported capabilities.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's core contributions are empirical system components with an explicitly stated modeling assumption, and no prediction reduces to a fitted input or self-citation chain.

full rationale

The paper does not present a mathematical derivation whose conclusion is equivalent to its premises. The central mechanism, Swin-DPM, is introduced as a sliding-window denoising construction: the paper explicitly states the finite-window assumption ("it becomes crucial to assume that temporal dependencies are confined within a limited time window") rather than deriving infinite-horizon coherence from it. The claimed 'infinite' extension follows algorithmically from appending new noise tokens after dequeuing denoised tokens, which is a system capability claim, not a fitted quantity renamed as a prediction. The quantitative evaluations (FVD, FID, CLIP, Move-PSNR, Move-LPIPS) are computed on generated outputs against held-out or ground-truth test videos; the text does not show that any metric is constructed from the same data used to fit a parameter that is then reported as the result. Table 1 is an author-assigned feature comparison, which is a comparative claim rather than a circular derivation. References used for the backbone, LoRA, and StreamDiffusion are external prior work, and there is no load-bearing self-citation chain or uniqueness theorem imported from the authors' own prior papers. The finite-window premise may be empirically unvalidated for hour-long coherence, and the claim of zero-shot generalization may be under-supported, but these are correctness risks, not circularity. Therefore no circular step is identified and the score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

All of the central capabilities rest on transferred assumptions from prior systems and on hand-chosen hyperparameters. There are no parameter-free derivations; the paper is an empirical system paper. The most important assumption is the finite temporal window in Swin-DPM, followed by transferability of Open-Sora's video prior to interactive control and the reliability of game-memory action extraction. No new physical entities are introduced.

free parameters (5)
  • causal relation range omega = 4
    Pre-defined to limit each keyboard control to influence only the current and next 4 video tokens. Chosen by hand; no ablation.
  • Swin-DPM window size w = w = T (default video token count)
    Window size for the sliding denoising queue. Set equal to T without sensitivity analysis; determines how much context the model sees during streaming.
  • default-text replacement probability q = 0.1
    Randomly replaces labeled keyboard inputs with 'The camera is moving in an unknown way' during training; hand-set robustness hyperparameter.
  • LoRA rank = 128
    Rank of low-rank adaptation in the warm-up stage; chosen without a reported sweep.
  • training steps per stage = 20k warm-up, 20k interactive, 60k Swin-DPM, 10k distillation
    Computation budget choices; no ablation or convergence analysis is reported for these step counts.
assumptions (5)
  • domain assumption Temporal dependencies in video are confined to a window of w tokens; attention beyond the window is unnecessary.
    This is the core premise of Swin-DPM, stated in Section 3.1. If long-range coherence matters, infinite streaming will drift.
  • domain assumption A pre-trained video DiT (Open-Sora, 2.3B) retains and transfers visual knowledge after LoRA and interactive-module fine-tuning.
    The whole approach depends on adapting a text-to-video model into an interactive world simulator rather than training from scratch.
  • domain assumption Unlabeled real-world and game footage improves visual quality and real-world generalization even without action labels.
    Warm-up and joint training use unlabeled data; no controlled experiment shows the marginal benefit of this data.
  • domain assumption Control signals captured by GameData are frame-aligned with video within at most three frames, and the delay is stable and subtractable.
    Appendix B.1 reports the three-frame bound based on observation, not a systematic calibration; frame-level control accuracy depends on this alignment.
  • standard math Standard video metrics (FVD, FID, CLIP, LPIPS, PSNR) and the chosen Move-LPIPS/Move-PSNR protocol capture interactive control quality.
    Metrics are adopted from prior work without validation against human judgment or downstream task performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control." pith.science (2026). https://pith.science/paper/MM4CP7IH

@misc{pith2026241203568,
  author       = {Pith},
  title        = {Pith review of: The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MM4CP7IH}},
  note         = {Machine review of arXiv:2412.03568}
}
read the original abstract

We present The Matrix, the first foundational realistic world simulator capable of generating continuous 720p high-fidelity real-scene video streams with real-time, responsive control in both first- and third-person perspectives, enabling immersive exploration of richly dynamic environments. Trained on limited supervised data from AAA games like Forza Horizon 5 and Cyberpunk 2077, complemented by large-scale unsupervised footage from real-world settings like Tokyo streets, The Matrix allows users to traverse diverse terrains -- deserts, grasslands, water bodies, and urban landscapes -- in continuous, uncut hour-long sequences. Operating at 16 FPS, the system supports real-time interactivity and demonstrates zero-shot generalization, translating virtual game environments to real-world contexts where collecting continuous movement data is often infeasible. For example, The Matrix can simulate a BMW X3 driving through an office setting--an environment present in neither gaming data nor real-world sources. This approach showcases the potential of AAA game data to advance robust world models, bridging the gap between simulations and real-world applications in scenarios with limited data.

Figures

Figures reproduced from arXiv: 2412.03568 by the authors.

Figure 1
Figure 1. The Matrix is a foundational realistic world simula￾tor capable of generating infinitely long 720p high-fidelity real￾scene video streams with real-time, precise moving control. Click to play with Adobe Acrobat Reader! The upper 1-minute demo may need flushing time. an initial step toward realizing this vision by developing a world model that enables neural networks to ‘dream’ vi￾sually authentic environments. The r… view at source ↗
Figure 2
Figure 2. The training process of The Matrix begins with a pre￾trained video DiT backbone. First, the Interactive Module is warmed up using Synthesized Observations of Unreal Rendered Contextual Environments data with unsupervised LoRA to make subsequent training focus on movement, not visuals. Then, we train the Interactive Module for precise frame-level control. Swin￾DPM enables infinite-length generation, and Stream Consis… view at source ↗
Figure 3
Figure 3. Main components of The Matrix. participating in denoising and ensuring continuity between different windows. The network of Swin-DPM is fine-tuned from a pre-trained DiT model. During training, we sample 2w video tokens, where w is the window size. We usually set w = T. The first w tokens are used solely for warming up Swin-DPM and do not participate in backpropagation; loss is computed only on the last w tokens. At… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The GameData Platform that creates the Source dataset. It uses CheatEngine to capture in-game status from CPU memory and filter out unsatisfactory frames, such as those with stuck characters or irregular movements. Reshade removes game UIs and HUDs to ensure a more con…
Figure 5
Figure 5. Figure 5: The results demonstrate frame-level precise control achieved by the Interactive Module across diverse scenes, weather conditions, [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Long worlds generation results by The Matrix. More examples are included in Supplementary Videos. are provided in Appendix Section C.1. Quantitative Results. The last two columns of Tab. 2 present the quantitative evaluation of interaction precision, using LPIPS and PS…
Figure 7
Figure 7. Figure 7: Generalization ability of The Matrix on unseen scenes and objects. 4.2. Infinete-Horizon World Generation Traditional world simulators focused on precise control of￾ten rely on small, auto-regressive generators trained from scratch to minimize the significant memory an…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multiplayer Interactive World Models with Representation Autoencoders

    cs.CV 2026-07 accept novelty 7.0 of 10

    A 5B-parameter latent diffusion model generates real-time four-player Rocket League matches conditioned on all players' actions, staying stable far beyond its training horizon.

  2. Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    Per-frame natural-language action prompts enable simultaneous multi-entity control and cross-entity action transfer in interactive video world models, outperforming discrete action-index interfaces.

  3. EgoSim: Egocentric World Simulator for Embodied Interaction Generation

    cs.CV 2026-04 conditional novelty 6.5 of 10

    EgoSim generates spatially consistent egocentric interaction videos by conditioning a video diffusion model on updatable 3D point-cloud states and action keypoints extracted at scale from monocular videos.

  4. Flow Equivariant World Models: Memory for Partially Observed Dynamic Environments

    cs.LG 2026-01 conditional novelty 6.0 of 10

    Flow equivariant world models use a latent memory that shifts with the agent and with inferred object motion, giving stable long-horizon prediction under partial observability.

  5. CustomX: Unified Character, Action, and Scene Customization in Video World Models

    cs.CV 2025-12 conditional novelty 6.0 of 10

    AniX generates controllable videos of a user-supplied character performing typed actions inside a user-supplied 3D scene by fine-tuning a pre-trained video generator on small locomotion datasets.

  6. ECHO: Ego-Centric modeling of Human-Object interactions

    cs.CV 2025-08 conditional novelty 6.0 of 10

    ECHO jointly predicts human pose, object trajectory, and contact from sparse head-and-wrist tracking using a tri-variate diffusion transformer, and reports the best egocentric human-object interaction reconstruction r...

  7. Precise Action-to-Video Generation Through Visual Action Prompts

    cs.CV 2025-08 conditional novelty 6.0 of 10

    Skeleton-based visual action prompts give precise, cross-domain action control for video generation of human and robot interactions.

  8. From Virtual Games to Real-World Play

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A chunk-wise video diffusion model trained on labeled game data plus unlabeled real footage transfers game-style control commands to real-world entities.

  9. Matrix-Game: Interactive World Foundation Model

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A 17B-parameter diffusion model generates controllable, physically consistent Minecraft video from a reference image and user actions, beating Oasis and MineWorld on a new benchmark.

  10. Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Hunyuan-GameCraft generates long, action-controlled game videos from a single image by unifying keyboard/mouse inputs into a continuous camera space and conditioning on mixed historical context.

  11. Video World Models with Long-term Spatial Memory

    cs.CV 2025-06 conditional novelty 6.0 of 10

    An autoregressive video world model with a persistent static point-cloud spatial memory and sparse episodic keyframes improves revisit consistency over point-cloud-conditioned baselines.

  12. Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Context-as-Memory conditions video generation on selected historical frames chosen by camera FOV overlap, improving scene consistency in long generated videos.

  13. Long-Context State-Space Video World Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A hybrid state-space and local-attention architecture gives autoregressive video diffusion models long-term spatial memory with constant per-frame inference cost, demonstrated on Maze and Minecraft.

  14. HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A staged pipeline generates layered, mesh-based 3D worlds from text or images by combining panoramic diffusion, semantic layer decomposition, and video-based expansion.

  15. VRAG: Learning World Models for Interactive Video Generation

    cs.CV 2025-05 unverdicted novelty 5.0 of 10

    VRAG improves long-horizon interactive video generation by conditioning autoregressive diffusion on retrieved historical frames and explicit global state, outperforming long-context baselines on the tested Minecraft a...

  16. Pre-Trained Video Generative Models as World Simulators

    cs.CV 2025-02 conditional novelty 5.0 of 10

    A lightweight action-conditioning module and a motion-reinforced loss convert pre-trained video generators into action-following world simulators that also speed up model-based reinforcement learning.

  17. Hunyuan-Game: Industrial-grade Intelligent Game Creation Model

    cs.CV 2025-05 reject novelty 4.0 of 10

    Tencent's Hunyuan-Game applies diffusion transformers to game asset creation across nine image and video generation tasks, with self-reported gains that are partly contradicted by its own evaluation table.

  18. A Survey of Interactive Generative Video

    cs.CV 2025-04 conditional novelty 4.0 of 10

    A survey that divides interactive generative video research into five modules: generation, control, memory, dynamics, and intelligence.

Reference graph

Works this paper leans on

47 extracted references · 19 canonical work pages · cited by 18 Pith papers

  1. [1]

    Diffusion for world modeling: Visual details matter in Atari

    Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kan- ervisto, Amos Storkey, Tim Pearce, and Franc ¸ois Fleuret. Diffusion for world modeling: Visual details matter in Atari. arXiv preprint arXiv:2405.12399, 2024. 2

    arXiv 2024

  2. [2]

    Hin- ton

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hin- ton. Layer normalization. arXiv preprint arXiv:1607.06450,

    arXiv

  3. [3]

    Ge- nie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Ge- nie: Generative interactive environments. In International Conference on Machine Learning, 2024. 2

    work page 2024

  4. [4]

    MuDreamer: Learn- ing predictive world models without reconstruction

    Maxime Burchi and Radu Timofte. MuDreamer: Learn- ing predictive world models without reconstruction. arXiv preprint arXiv:2405.15083, 2024. 2

    arXiv 2024

  5. [5]

    GameGen-X: Interactive open-world game video generation

    Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. GameGen-X: Interactive open-world game video generation. arXiv preprint arXiv:2411.00769, 2024. 2

    arXiv 2024

  6. [6]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE conference on computer vision and pat- tern recognition, 2024. 12

    work page 2024

  7. [7]

    Oasis: A universe in a Transformer

    Decart, Julian Quevedo, Quinn McIntyre, Spruce Campbell, Xinlei Chen, and Robert Wachen. Oasis: A universe in a Transformer. 2024. 2

    work page 2024

  8. [8]

    Ffmpeg, 2023

    FFmpeg Developers. Ffmpeg, 2023. Accessed: 2023-10-01. 12

    work page 2023

Show all 47 references
  1. [9]

    Sigmoid- weighted linear units for neural network function approxima- tion in reinforcement learning

    Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid- weighted linear units for neural network function approxima- tion in reinforcement learning. Neural networks, 107:3–11,

  2. [10]

    https://www.cheatengine.org/ ,

    Cheat Engine. https://www.cheatengine.org/ ,

  3. [11]

    Recurrent world models facilitate policy evolution

    David Ha and J ¨urgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018. 2

    2018

  4. [12]

    Dream to control: Learning behaviors by la- tent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Moham- mad Norouzi. Dream to control: Learning behaviors by la- tent imagination. arXiv preprint arXiv:1912.01603, 2019. 2

    1912 arXiv

  5. [13]

    Mastering Atari with discrete world models

    Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering Atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020. 2

    2010 arXiv

  6. [14]

    Mastering diverse domains through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023. 2

    2023 arXiv

  7. [15]

    Temporal dif- ference learning for model predictive control

    Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal dif- ference learning for model predictive control. arXiv preprint arXiv:2203.04955, 2022. 2

    2022 arXiv

  8. [16]

    TD-MPC2: Scalable, robust world models for continuous control

    Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16828, 2023. 2

    2023 arXiv

  9. [17]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. 3, 10

    2016 arXiv

  10. [18]

    GANs trained by a two time-scale update rule converge to a local nash equi- librium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equi- librium. Advances in neural information processing systems, 30, 2017. 6

    2017

  11. [19]

    GAIA-1: A generative world model for au- tonomous driving

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gian- luca Corrado. GAIA-1: A generative world model for au- tonomous driving. arXiv preprint arXiv:2309.17080, 2023. 2

    2023 arXiv

  12. [20]

    LoRA: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021. 4, 10

    2021 arXiv

  13. [21]

    SafeDreamer: Safe reinforcement learn- ing with world models

    Weidong Huang, Jiaming Ji, Borong Zhang, Chunhe Xia, and Yaodong Yang. SafeDreamer: Safe reinforcement learn- ing with world models. arXiv preprint arXiv:2307.07176 ,

    arXiv

  14. [22]

    GPT-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. GPT-4o system card. arXiv preprint arXiv:2410.21276, 2024. 5

    2024 arXiv

  15. [23]

    Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models

    Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Min- jia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309.14509, 2023. 10

    2023 arXiv

  16. [24]

    DROID: A large-scale in-the-wild robot manipulation dataset

    Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ash- win Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yun- liang Chen, Kirsty Ellis, et al. DROID: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:24...

    2024 arXiv

  17. [25]

    Auto-encoding variational Bayes

    Diederik P Kingma. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114, 2013. 5

    2013 arXiv

  18. [26]

    Streamdiffu- sion: A pipeline-level solution for real-time interactive gen- eration

    Akio Kodaira, Chenfeng Xu, Toshiki Hazama, Takanori Yoshimoto, Kohei Ohno, Shogo Mitsuhori, Soichi Sugano, Hanying Cho, Zhijian Liu, and Kurt Keutzer. Streamdiffu- sion: A pipeline-level solution for real-time interactive gen- eration. arXiv preprint arXiv:2312.12491, 2023. 5

    2023 arXiv

  19. [27]

    Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 14

    2023 arXiv

  20. [28]

    Latent consistency models: Synthesizing high- resolution images with few-step inference, 2023

    Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high- resolution images with few-step inference, 2023. 10

    2023

  21. [29]

    Sora: Creating video from text

    OpenAI. Sora: Creating video from text. Preprint, 2024. 2

    2024

  22. [30]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023. 2, 3

    2023

  23. [31]

    Video game generation: A practical study using Mario

    Virtuals Protocol. Video game generation: A practical study using Mario. Preprint, 2024. 2

    2024

  24. [32]

    Learning 18 transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning 18 transferable visual models from natural language supervi- sion. In International conference on machine learnin...

    2021

  25. [33]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 4

    2020

  26. [34]

    https://obsproject.com/ , 2024

    OBS Recording. https://obsproject.com/ , 2024. Software. 5

    2024

  27. [35]

    https://reshade.me/, 2024

    Reshade. https://reshade.me/, 2024. Software. 5

    2024

  28. [36]

    On learning to think: Algorithmic information theory for novel combinations of reinforce- ment learning controllers and recurrent neural world models

    J ¨urgen Schmidhuber. On learning to think: Algorithmic information theory for novel combinations of reinforce- ment learning controllers and recurrent neural world models. arXiv preprint arXiv:1511.09249, 2015. 2

    2015 arXiv

  29. [37]

    Mastering Atari, Go, chess and shogi by planning with a learned model

    Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020. 2

    2020

  30. [38]

    Improved techniques for training consistency models

    Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In International Conference on Learning Representations, 2024. 4

    2024

  31. [39]

    Consistency models

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023. 4

    2023 arXiv

  32. [40]

    Image pro- cessing, analysis and machine vision

    Milan Sonka, Vaclav Hlavac, and Roger Boyle. Image pro- cessing, analysis and machine vision. Springer, 2013. 6

    2013

  33. [41]

    To- wards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. 6

    2018 arXiv

  34. [42]

    Diffusion models are real-time game engines

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837, 2024. 2

    2024 arXiv

  35. [43]

    DayDreamer: World models for physical robot learning

    Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. DayDreamer: World models for physical robot learning. In Conference on robot learning , pages 2226–2240. PMLR, 2023. 2

    2023

  36. [44]

    Pandora: Towards general world model with natural language actions and video states

    Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, et al. Pandora: Towards general world model with natural language actions and video states. arXiv preprint arXiv:2406.09455, 2024. 2

    2024 arXiv

  37. [45]

    Learn- ing interactive real-world simulators

    Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learn- ing interactive real-world simulators. arXiv preprint arXiv:2310.06114, 2023. 2

    2023 arXiv

  38. [46]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 6

    2018

  39. [47]

    Open-Sora: Democratizing efficient video production for all, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-Sora: Democratizing efficient video production for all, 2024. 3, 10 19

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.