Pith. sign in

REVIEW 3 major objections 7 minor 15 cited by

Matrix-Game: Interactive World Foundation Model

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read Matrix-Game claims that a 17B-parameter image-to-world diffusion model generates Minecraft videos that follow keyboard and mouse commands with 95% accuracy and that it outperforms the open-source baselines Oasis and MineWorld on every…

desk verdict A serious large-scale contribution, but the controllability numbers rest on an unvalidated IDM metric and need ablations, error bars, and artifact release before being trusted. read the letter →

arxiv 2506.18701 v1 pith:V2T64JEV submitted 2025-06-23 cs.CV cs.AI

classification cs.CVcs.AI
keywords Minecraftworldgenerationinteractivemodelvideodiffusiontransformeraction-conditionedcontrollablebenchmarkinversedynamicsimage-to-world
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that Matrix-Game, a 17B-parameter video diffusion model, can turn a single reference image of a Minecraft scene into a long, controllable video: it follows keyboard commands (forward, back, jump, attack) and mouse-driven camera movements, while keeping the scene and objects physically consistent. This is achieved by a two-stage training pipeline: first, large-scale unlabeled gameplay video teaches the model the look and dynamics of the Minecraft world; second, action-labeled clips teach it to respond to user control. The authors also introduce a benchmark, GameWorld Score, that measures eight properties of generated worlds, and report that Matrix-Game outperforms the open-source models Oasis and MineWorld on all of them, with the largest gains in control accuracy and physical consistency. If true, this would be a step toward using generative models as interactive game engines that a user can explore and steer from a single starting frame.

What carries the argument

The load-bearing design is an action-conditioned latent diffusion transformer that generates video in a spatiotemporally compressed latent space built by a 3D causal VAE. The model concatenates the latent of the last five generated frames as motion context with a binary mask, applies a token-replacement trick inherited from prior image-to-video work for stable image-to-video generation, and injects control signals through a module that applies the group-operation trick to align action tokens with temporally compressed latents: keyboard actions enter through cross-attention, mouse movements through an MLP plus temporal self-attention. Training uses the rectified flow loss in two stages—unlabeled pretraining on 2,700 hours of filtered gameplay, then action-labeled training on about 1,200 hours—and autoregressive long-video generation is achieved by chaining segments with noise and classifier-free guidance on the motion context.

What would settle it

Take Table 2's action-controllability test and replace the IDM with an action estimator trained exclusively on synthetic Minecraft-like frames, then re-measure keyboard and mouse accuracy on the same generated clips; if accuracy drops substantially below 0.95 while the videos look unchanged, the reported control figures were inflated by IDM blind spots.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that an image-to-world latent diffusion transformer, scaled to 17B parameters and trained first without labels and then with action labels, can produce visually and temporally coherent Minecraft videos that accurately mirror the user's keyboard and mouse inputs. The model conditions only on a reference image and motion context, not on text, and employs an action-control module: discrete keyboard actions are injected through cross-attention, continuous mouse angles through temporal self-attention, with classifier-free guidance applied to actions, motion frames, and the reference image. Evaluated by the paper's GameWorld Score, the model reports 0.95 keyboard accuracy and 0.95 mouse accuracy, versus 0.86 and 0.64 for the best prior open-source baseline (MineWorld), and the authors report higher object and scenario consistency.

Load-bearing premise

The reported control accuracy assumes that the Inverse Dynamics Model trained on 1,962 hours of real Minecraft gameplay gives valid action labels when applied to generated video; if the generated frames exploit statistical regularities the IDM has memorized, the 95% keyboard and mouse accuracies would overstate true control.

Editorial extensions

If this is right

  • If the reported accuracies hold, interactive video generation from a single image becomes a practical way to explore and steer open-ended game worlds without a hand-written game engine.
  • The two-stage training recipe—unlabeled world pretraining followed by action-labeled fine-tuning—becomes a default template for building controllable world models in other games and simulated environments.
  • The GameWorld Score benchmark supplies a standardized set of eight measurements that future Minecraft world models can be compared against.
  • Autoregressive chaining of action-conditioned clips with a five-frame motion context extends controllability to arbitrarily long videos while keeping temporal coherence across segment boundaries.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The IDM-based controllability metric may overestimate true control: if the IDM was trained on the same visual style of Minecraft footage, it could be exploiting appearance-level regularities rather than verifying that the scene actually responds to the action semantics; a human-in-the-loop test where users issue commands to a live model would be a stronger check.
  • The method's reliance on Minecraft-specific unlabeled data and IDM labels means the generalization claim to other games (such as racing or combat simulators) is, for now, a roadmap rather than a demonstrated result.
  • A testable extension would be to measure whether the same two-stage pipeline with a separate, non-Minecraft dataset (for example, a physics sandbox) yields similar control accuracy, which would separate the contribution of the architecture from the contribution of the Minecraft-specific data.
  • If GameWorld Score becomes standard, its scenario-consistency test—which pairs symmetric camera motions and requires the scene to reappear—could serve as a useful proxy for whether a world model truly reasons about a persistent 3D scene rather than a bag of frames.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes Matrix-Game, a 17B-parameter latent diffusion model for interactive Minecraft world generation, conditioned on a reference image, motion context, and keyboard/mouse actions. It introduces a two-stage training pipeline and the Matrix-Game-MC dataset (about 2,700 hours of unlabeled video and 1,000–1,200 hours of action-labeled video), as well as GameWorld Score, an eight-metric benchmark covering visual quality, temporal quality, action controllability, and physical consistency. The authors report that Matrix-Game outperforms Oasis and MineWorld on all benchmark dimensions, with keyboard and mouse accuracies of 0.95, and that double-blind human evaluation confirms its superiority. They plan to release model weights and benchmark code.

Significance. If the claims hold, this is a substantial open-source contribution to interactive world modeling, combining a large-scale dataset, a two-stage training scheme, and a multi-dimensional evaluation suite. The paper is honest about failure cases and remaining limitations, which is a strength. However, the central quantitative evidence for controllability and the headline comparison rests on a newly proposed benchmark whose action-controllability component may be confounded with the action-labeling pipeline used to train the data, and the paper does not report error bars, evaluation-set details, or validation of the IDM on generated video. The human study supports the direction but is under-reported. Because the method and dataset are plausible and the issues are addressable through additional analysis, the manuscript merits major revision rather than rejection.

major comments (3)
  1. [§5.3, Tables 2–3] The action-controllability pillar labels generated videos with the same class of Inverse Dynamics Model (IDM) that is embedded elsewhere in the data pipeline: Section 3.1 uses an IDM for menu-state and camera-movement filtering, and Section 3.2 constructs action labels via VPT agents, which rely on IDM-style action inference. The benchmark's validity therefore depends on the IDM's precision on synthetic video, which the paper does not report. I ask for (i) a statement of whether the evaluation IDM is the same checkpoint as the one used in data creation; (ii) a calibration experiment reporting IDM accuracy on generated videos with known ground-truth actions, split by scenario; and (iii) a comparison of IDM-based controllability scores with human judgments of action following on the same clips. This is load-bearing because Tables 2 and 3 are the primary evidence for the central claim of precise control; the failure cases in Section 6.5 show that physically implausible generations can still receive high IDM-based accuracy, confirming that the metric may reward style mimicry rather than true action following.
  2. [§6.1, Table 2 and Figure 8] The headline comparisons are reported as point estimates without error bars, number of evaluation videos, seeds, or significance tests, so it is impossible to judge whether the margins over Oasis and MineWorld are meaningful. The evaluation set should be specified: how many reference images, action sequences, clip lengths, and generation seeds per model, and whether the same reference images and action sequences were used for all models. The human study in Figure 8 reports only aggregate win rates; please provide the number of annotators, number of comparisons per dimension, instructions, inter-annotator agreement, and significance of the win-rate differences. Without this information, the claim that Matrix-Game 'consistently outperforms' the baselines is not fully supported.
  3. [§3.2 and §4.2] The labeled dataset size is given inconsistently: Section 3.2 reports 'over 1,026 hours of video clips for 33-frame training' and separately 'more than 1,200 hours of video for 65-frame training,' whereas Section 4.2 Stage 2 states the model is 'trained on 1,200 hours of action-labeled 720p 33-frame video clips' and then later uses 'approximately 1,200 hours of 720p 65-frame video clips.' Please clarify which dataset was used in each training stage and reconcile the 1,026-hour and 1,200-hour numbers; if two different labeled sets were used, state the order and how they relate.
minor comments (7)
  1. [§5.3] The mouse-direction evaluation uses a 'pre-defined threshold' for detecting rotational change, but the threshold value is never given; please specify it and describe how continuous predicted yaw and pitch are mapped to the eight directional categories.
  2. [§5.3] The IDM used for evaluation is described only as 'trained on 1,962 hours of Minecraft gameplay' with no citation or checkpoint identifier; please state whether it is the VPT IDM of [4], a MineWorld [18] release, or a new model, and provide the source.
  3. [§6.1, Figure 8] Calling the human study 'double-blind' is imprecise: the annotators are blind to method identity, but the selection of videos and their pairing are not described; please specify who selected the videos, how many were used per condition, and the presentation order and randomization.
  4. [§2.3] In the sentence listing prior game-video-generation works, reference [41] appears twice; please remove the duplicate.
  5. [§6.2, Table 3] The text says the model attains 'over 88% accuracy on all actions,' but the jump accuracy in Table 3 is exactly 0.88; either rephrase to 'at least 88%' or correct the number.
  6. [§4.1, §5.3, §7] The paper states in Section 4.1 that mouse movements are continuous scalar values, but Section 5.3 evaluates only eight directions, and Section 7 lists 'mouse control with a broader and more continuous value range' as future work; please clarify whether the trained model already supports continuous mouse input and why the benchmark only tests discrete directions.
  7. [Abstract and §5] The paper announces that code and weights will be open-sourced, but provides no link, license, or release timeline; please add concrete availability information or state that the release is pending final approval.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the claimed controllability gains are post-hoc measurements with an external IDM and independent human evaluation, not fitted predictions.

full rationale

I walked the paper's derivation chain and found no step in which a prediction or claimed result reduces by construction to its inputs. Matrix-Game is trained with a rectified-flow loss on unlabeled and action-labeled video (Sections 3–4); the keyboard/mouse accuracies in Tables 2–3 are computed after training by GameWorld Score, whose action-controllability pillar uses an Inverse Dynamics Model adopted from MineWorld/VPT (Section 5.3) to label generated videos. The IDM is not trained by the authors on Matrix-Game outputs, and no parameter of Matrix-Game is fitted to the benchmark, so the 0.95/0.95 numbers are measurements rather than fit-derived predictions. The IDM is also used in data curation (menu-state and camera-movement filtering, Section 3.1), but that only weakens the metric's independence; it does not make the model's control accuracy true by definition. The human double-blind evaluation (Figure 8) and the external baselines Oasis and MineWorld provide independent support for the ranking. Self-citations appear only in routine methodological contexts (e.g., references [8, 34, 69–73] for augmentation and long-tailed training) and are not load-bearing for the central claim. The admitted failure cases in Section 6.5 (e.g., walking through leaves) qualify the physical-consistency claim but are limitations, not circularity. Therefore no circular step is identified.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper contributes no new physical entities or forces. The main postulates are the validity of external pretrained evaluators and the transferability of data sources, which are domain assumptions rather than free parameters.

free parameters (3)
  • per-frame yaw/pitch rotation cap in labeled data creation = 15 degrees
    Hand-set to stabilize camera motion; directly shapes the controllability supervision and the distribution of mouse-action labels.
  • IDM mouse-movement detection threshold = not specified
    Predefined threshold separates directional camera changes from 'empty'; changes to this threshold change Mouse Acc results.
  • CFG scales and probabilities = CFG 6 for actions/motion/reference; action dropout 0.1; motion dropout 0.25
    Chosen by hand for training and inference; not swept or justified in the paper.
assumptions (4)
  • domain assumption IDM trained on 1,962 hours of Minecraft gameplay is a valid action oracle for generated video
    Used in Section 5.3 to measure keyboard and mouse controllability; if the IDM does not transfer to synthetic frames, all controllability numbers are uninterpretable.
  • domain assumption Unreal Engine procedural videos transfer to Minecraft dynamics
    Stage 2 training mixes Unreal sequences with Minecraft data; the paper assumes this aids Minecraft generation without a dedicated transfer experiment.
  • domain assumption Pretrained HunyuanVideo weights provide a useful visual prior after replacing the text branch
    Initialization from HunyuanVideo is stated in Section 4.2; no ablation measures how much of the final quality comes from this prior.
  • domain assumption Laion aesthetic, MUSIQ, CLIP, AMT, and DROID-SLAM are valid no-reference metrics for game video
    GameWorld Score relies on these external predictors; the paper does not recalibrate or validate them on Minecraft-specific artifacts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Matrix-Game: Interactive World Foundation Model." pith.science (2026). https://pith.science/paper/V2T64JEV

@misc{pith2026250618701,
  author       = {Pith},
  title        = {Pith review of: Matrix-Game: Interactive World Foundation Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V2T64JEV}},
  note         = {Machine review of arXiv:2506.18701}
}
read the original abstract

We introduce Matrix-Game, an interactive world foundation model for controllable game world generation. Matrix-Game is trained using a two-stage pipeline that first performs large-scale unlabeled pretraining for environment understanding, followed by action-labeled training for interactive video generation. To support this, we curate Matrix-Game-MC, a comprehensive Minecraft dataset comprising over 2,700 hours of unlabeled gameplay video clips and over 1,000 hours of high-quality labeled clips with fine-grained keyboard and mouse action annotations. Our model adopts a controllable image-to-world generation paradigm, conditioned on a reference image, motion context, and user actions. With over 17 billion parameters, Matrix-Game enables precise control over character actions and camera movements, while maintaining high visual quality and temporal coherence. To evaluate performance, we develop GameWorld Score, a unified benchmark measuring visual quality, temporal quality, action controllability, and physical rule understanding for Minecraft world generation. Extensive experiments show that Matrix-Game consistently outperforms prior open-source Minecraft world models (including Oasis and MineWorld) across all metrics, with particularly strong gains in controllability and physical consistency. Double-blind human evaluations further confirm the superiority of Matrix-Game, highlighting its ability to generate perceptually realistic and precisely controllable videos across diverse game scenarios. To facilitate future research on interactive image-to-world generation, we will open-source the Matrix-Game model weights and the GameWorld Score benchmark at https://github.com/SkyworkAI/Matrix-Game.

Figures

Figures reproduced from arXiv: 2506.18701 by the authors.

Figure 1
Figure 1. Controllable world generation results of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Model performance under our GameWorld Score benchmark, covering 8 key dimensions: [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. We construct our high-quality unlabeled training data from raw gameplay videos via a [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Overview of the interactive image-to-world generation paradigm. The model is trained in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: (a) Autoregressive generation in Matrix-Game and (b) The architecture of Matrix-Game. To [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: The details of diffusion transformer blocks in Matrix-Game. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: GameWorld Score provides a unified benchmark for assessing the quality and realism of [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Human evaluation results (double-blind setting) across four predefined dimensions: Overall [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: GameWorld Score across eight scenarios. Each radar chart shows performance over 8 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Auto-regressive generation results of Matrix-Game across three action-conditioned segments. Each segment is generated independently based on the preceding motion context and current action signal. Despite the segment-wise generation, Matrix-Game maintains strong tempo…
Figure 11
Figure 11. Figure 11: Failure cases of Matrix-Game. (a) Edge case: the model may fail to maintain temporal consistency in underrepresented or unfamiliar scenarios. (b) Physics understanding: the agent walks through leaves, indicating that there is room to improve the modeling of physical i…
Figure 12
Figure 12. Figure 12: Demonstration of Matrix-Game’s controllable video generation conditioned on diverse mouse movement commands, including horizontal (left, right), vertical (up, down), and diagonal camera adjustments. The model responds accurately to subtle changes in camera direction a…
Figure 13
Figure 13. Figure 13: Demonstration of Matrix-Game’s controllable video generation conditioned on various keyboard actions, including forward, back, left, right, jump, and attack. The model accurately responds to user’s control signals and generates coherent motion patterns. 24 [PITH_FULL…
Figure 14
Figure 14. Figure 14: Demonstration of Matrix-Game’s controllable video generation conditioned on complex actions, such as forward + right, backward + attack and jump + attack,. The model effectively interprets complex user commands and generates coherent, action-consistent motion trajecto…
Figure 15
Figure 15. Figure 15: Controllable world generation results of [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    Per-frame natural-language action prompts enable simultaneous multi-entity control and cross-entity action transfer in interactive video world models, outperforming discrete action-index interfaces.

  2. ActionParty: Multi-Subject Action Binding in Generative Video Games

    cs.CV 2026-04 conditional novelty 7.0 of 10

    ActionParty binds discrete actions to individual subjects in a single generated video by jointly modeling subject state tokens and video latents, controlling up to seven players across 46 Melting Pot games.

  3. RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation

    cs.RO 2026-07 unverdicted novelty 6.5 of 10

    Step Forcing trains a few-step autoregressive video world model so RoboWorld closed-loop rollouts plus a task-progress VLM judge recover real-world policy rankings at r=0.989 and ρ=0.970.

  4. Alaya-EVOKE: From Linear-Scaling Supervision to Endless World

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Evoke combines an external camera-indexed geometry memory with a long-horizon teacher to produce a three-step world model that generates hour-scale video with bounded per-step cost.

  5. WorldCycle: Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models

    cs.AI 2026-08 conditional novelty 6.0 of 10

    WorldCycle post-trains interactive video world models with reinforcement learning rewards for spatial closure and temporal consistency on reversible action cycles, reducing long-horizon drift and improving composite-a...

  6. Wonder: Video World Model Done Better

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Wonder generates minute-scale, real-time camera-controllable video worlds from a single image or video at 16 FPS, using a rendered coordinate-field control signal, sparse full-fidelity memory, and stage-specialized di...

  7. DynaVieW: Schema-Guided World Modeling for Understanding Hierarchical Visual Dynamics

    cs.LG 2026-07 accept novelty 6.0 of 10

    Schema-guided interleaved state-transition pretraining with selective attention and reweighted loss improves hierarchical visual dynamics modeling for narrative generation and world simulation.

  8. Worldscape-MoE: A Unified Mixture-of-Experts World Model for Scalable Heterogeneous Action Control

    cs.RO 2026-07 conditional novelty 6.0 of 10

    A DiT Mixture-of-Experts world model jointly learns locomotion, dual-arm manipulation, and egocentric hand control, with shared experts for world dynamics and progressive expert expansion for new modalities.

  9. WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL

    cs.RO 2026-02 conditional novelty 6.0 of 10

    WoVR shows that reinforcement learning can improve VLA robot policies through imagined rollouts in a video world model, reporting +29.3 points on LIBERO and +30.0 points on real Franka tasks.

  10. CustomX: Unified Character, Action, and Scene Customization in Video World Models

    cs.CV 2025-12 conditional novelty 6.0 of 10

    AniX generates controllable videos of a user-supplied character performing typed actions inside a user-supplied 3D scene by fine-tuning a pre-trained video generator on small locomotion datasets.

  11. End-to-End Training for Autoregressive Video Diffusion via Self-Resampling

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Resampling Forcing trains autoregressive video diffusion models on self-resampled degraded histories with a causal mask, achieving stable long-horizon generation without a teacher or discriminator.

  12. Quo Vadis, World Modeling?

    cs.CV 2026-08 conditional novelty 5.0 of 10

    An agent-centric reframing of world modeling, replacing physical state prediction with 'information transitions' organized into six proxy functions and three empowerment levels.

  13. From Pixels to States: Rethinking Interactive World Models as Game Engines

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Interactive world models are reorganized around the game-engine action-state-observation loop, and a 90-hour Black Myth: Wukong dataset with frame-aligned actions, ground-truth states, and observations is introduced.

  14. Resilience Revisited: A Multidimensional Framework Derived from Realistic Attack Scenarios

    eess.SY 2026-04 unverdicted novelty 5.0 of 10

    A new MDRI framework decomposes power system resilience into five dimensions with a calibrated multiplicative term, showing multi-vector attacks cause 5.6 times more degradation than linear sums plus 60.6% from coupling.

  15. OpenWorldLib: A Unified Codebase and Definition of Advanced World Models

    cs.CV 2026-04 unverdicted novelty 4.0 of 10

    OpenWorldLib offers a standardized codebase and definition for world models that combine perception, interaction, and memory to understand and predict the world.

Reference graph

Works this paper leans on

74 extracted references · 34 canonical work pages · cited by 15 Pith papers

  1. [1]

    Cosmos: world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575, 2025

    Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos: world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575, 2025

  2. [2]

    Diffusion for world modeling: Visual details matter in atari

    Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos J Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. InAdvances in Neural Information Processing Systems, volume 37, pages 58757–58791, 2024

  3. [3]

    Character region awareness for text detection

    Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee. Character region awareness for text detection. InComputer Vision and Pattern Recognition, pages 9365–9374, 2019

  4. [4]

    Video pretraining (vpt): Learning to act by watching unlabeled online videos

    Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. InAdvances in Neural Information Processing Systems, volume 35, pages 24639–24654, 2022

  5. [5]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

  6. [6]

    Genie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. InInternational Conference on Machine Learning, 2024

  7. [7]

    Gamegen-x: Interactive open-world game video generation.arXiv preprint arXiv:2411.00769, 2024

    Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamegen-x: Interactive open-world game video generation.arXiv preprint arXiv:2411.00769, 2024

  8. [8]

    One-dm: One-shot diffusion mimicker for handwritten text generation

    Gang Dai, Yifan Zhang, Quhui Ke, Qiangya Guo, and Shuangping Huang. One-dm: One-shot diffusion mimicker for handwritten text generation. InEuropean Conference on Computer Vision, pages 410–427. Springer, 2024

Show all 74 references
  1. [9]

    Oasis: A universe in a transformer

    Decart. Oasis: A universe in a transformer. 2024

  2. [10]

    Worldscore: A unified evaluation benchmark for world generation.arXiv preprint arXiv:2504.00983, 2025

    Haoyi Duan, Hong-Xing Yu, Sirui Chen, Li Fei-Fei, and Jiajun Wu. Worldscore: A unified evaluation benchmark for world generation.arXiv preprint arXiv:2504.00983, 2025

  3. [11]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. InInternational Conference on Machine Learning, 2024

  4. [12]

    Minedojo: Building open-ended embodied agents with internet-scale knowledge

    Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. InAdvances in Neural Information Processing Systems, volume 35...

  5. [13]

    Perceptual quality assessment of smartphone photography

    Yuming Fang, Hanwei Zhu, Yan Zeng, Kede Ma, and Zhou Wang. Perceptual quality assessment of smartphone photography. InComputer Vision and Pattern Recognition, 2020

  6. [14]

    The matrix: Infinite-horizon world generation with real-time moving control

    Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, and Hongyang Zhang. The matrix: Infinite-horizon world generation with real-time moving control. arXiv preprint arXiv:2412.03568, 2024

  7. [15]

    FFmpeg: A complete, cross-platform solution to record, convert and stream audio and video.https://ffmpeg.org/, 2024

    FFmpeg Team. FFmpeg: A complete, cross-platform solution to record, convert and stream audio and video.https://ffmpeg.org/, 2024

  8. [16]

    Vista: A generalizable driving world model with high fidelity and versatile controllability

    Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. arXiv preprint arXiv:2405.17398, 2024

  9. [17]

    Long-context autoregressive video modeling with next-frame prediction.arXiv preprint arXiv:2503.19325, 2025

    Yuchao Gu, weijia Mao, and Mike Zheng Shou. Long-context autoregressive video modeling with next-frame prediction.arXiv preprint arXiv:2503.19325, 2025. 19

  10. [18]

    Mineworld: a real-time and open-source interactive world model on minecraft.arXiv preprint arXiv:2504.08388, 2025

    Junliang Guo, Yang Ye, Tianyu He, Haoyu Wu, Yushu Jiang, Tim Pearce, and Jiang Bian. Mineworld: a real-time and open-source interactive world model on minecraft.arXiv preprint arXiv:2504.08388, 2025

  11. [19]

    The essential role of causality in foundation world models for embodied ai.arXiv preprint arXiv:2402.06665, 2024

    Tarun Gupta, Wenbo Gong, Chao Ma, Nick Pawlowski, Agrin Hilmkil, Meyer Scetbon, Marc Rigter, Ade Famoti, Ashley Juan Llorens, Jianfeng Gao, et al. The essential role of causality in foundation world models for embodied ai.arXiv preprint arXiv:2402.06665, 2024

  12. [20]

    World models.arXiv preprint arXiv:1803.10122, 2018

    David Ha and Jürgen Schmidhuber. World models.arXiv preprint arXiv:1803.10122, 2018

  13. [21]

    PhD thesis, University of Toronto (Canada), 2024

    Danijar Hafner.Embodied Intelligence Through World Models. PhD thesis, University of Toronto (Canada), 2024

  14. [22]

    Cameractrl: Enabling camera control for text-to-video generation.arXiv preprint arXiv:2404.02101, 2024

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation.arXiv preprint arXiv:2404.02101, 2024

  15. [23]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. InAdvances in Neural Information Processing Systems, volume 33, pages 6840–6851, 2020

  16. [24]

    Gaia-1: A generative world model for autonomous driving.arXiv preprint arXiv:2309.17080, 2023

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving.arXiv preprint arXiv:2309.17080, 2023

  17. [25]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. InComputer Vision and Pattern Recognition, pages 21807–21818, 2024

  18. [26]

    MUSIQ: multi-scale image quality transformer.arXiv preprint arXiv:2108.05997, 2021

    Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. MUSIQ: multi-scale image quality transformer.arXiv preprint arXiv:2108.05997, 2021

  19. [27]

    Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

  20. [28]

    Auto-encoding variational bayes, 2013

    Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013

  21. [29]

    Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

  22. [30]

    Flux.https://github.com/black-forest-labs/flux, 2024

    Black Forest Labs. Flux.https://github.com/black-forest-labs/flux, 2024

  23. [31]

    aesthetic-predictor.https://github.com/LAION-AI/aesthetic-predictor, 2022

    LAION-AI. aesthetic-predictor.https://github.com/LAION-AI/aesthetic-predictor, 2022

  24. [32]

    Amt: All-pairs multi-field transforms for efficient frame interpolation

    Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. Amt: All-pairs multi-field transforms for efficient frame interpolation. InComputer Vision and Pattern Recognition, 2023

  25. [33]

    Open-sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024

    Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024

  26. [34]

    Drivegen: Generalized and robust 3d detection in driving via controllable text-to-image diffusion generation

    Hongbin Lin, Zilu Guo, Yifan Zhang, Shuaicheng Niu, Yafeng Li, Ruimao Zhang, Shuguang Cui, and Zhen Li. Drivegen: Generalized and robust 3d detection in driving via controllable text-to-image diffusion generation. InComputer Vision and Pattern Recognition, 2025

  27. [35]

    World model on million-length video and language with ringattention.arXiv e-prints, pages arXiv–2402, 2024

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention.arXiv e-prints, pages arXiv–2402, 2024

  28. [36]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. InInternational Conference on Learning Representations, 2023

  29. [37]

    A survey on vision-language-action models for embodied ai.arXiv preprint arXiv:2405.14093, 2024

    Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. A survey on vision-language-action models for embodied ai.arXiv preprint arXiv:2405.14093, 2024

  30. [38]

    MineRL: A large-scale dataset of minecraft demonstrations

    MineRL Project. MineRL: A large-scale dataset of minecraft demonstrations. https://minerl. readthedocs.io/en/latest/, 2020

  31. [39]

    Conditional image-to- video generation with latent flow diffusion models

    Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image-to- video generation with latent flow diffusion models. InComputer Vision and Pattern Recognition, pages 18444–18455, 2023

  32. [40]

    Sora: Video generation models as world simulators

    OpenAI. Sora: Video generation models as world simulators. https://openai.com/index/ video-generation-models-as-world-simulators/, 2024

  33. [41]

    Genie 2: A large-scale foundation world model.URL: https://deepmind

    J Parker-Holder, P Ball, J Bruce, V Dasagi, K Holsheimer, C Kaplanis, A Moufarek, G Scully, J Shar, J Shi, et al. Genie 2: A large-scale foundation world model.URL: https://deepmind. google/discover/blog/genie- 2-a-large-scale-foundation-world-model, 2024

  34. [42]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InInternational Conference on Computer Vision, pages 4195–4205, 2023. 20

  35. [43]

    SDXL: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, 2023

  36. [44]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning, 2021

  37. [45]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InComputer Vision and Pattern Recognition, pages 10684– 10695, 2022

  38. [46]

    A benchmark of facial recognition pipelines and co-usability perfor- mances of modules.Journal of Information Technologies, 17(2):95–107, 2024

    Sefik Serengil and Alper Ozpinar. A benchmark of facial recognition pipelines and co-usability perfor- mances of modules.Journal of Information Technologies, 17(2):95–107, 2024

  39. [47]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InInternational Conference on Machine Learning, pages 2256–2265. PMLR, 2015

  40. [48]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. InInternational Conference on Learning Representations, 2021

  41. [49]

    Transnet v2: An effective deep network architecture for fast shot transition detection

    Tomás Soucek and Jakub Lokoc. Transnet v2: An effective deep network architecture for fast shot transition detection. InACM International Conference on Multimedia, pages 11218–11221, 2024

  42. [50]

    Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras

    Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. In Advances in Neural Information Processing Systems, volume 34, pages 16558–16569, 2021

  43. [51]

    Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. InEuropean Conference on Computer Vision, pages 244–260. Springer, 2024

  44. [52]

    Diffusion models are real-time game engines.arXiv preprint arXiv:2408.14837, 2024

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines.arXiv preprint arXiv:2408.14837, 2024

  45. [53]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems, volume 30, 2017

  46. [54]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

  47. [55]

    Drivedreamer: Towards real-world-drive world models for autonomous driving

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world-drive world models for autonomous driving. InEuropean Conference on Computer Vision, pages 55–72, 2024

  48. [56]

    Motionctrl: A unified and flexible motion controller for video generation

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. InACM SIGGRAPH, pages 1–11, 2024

  49. [57]

    Generating worlds.https://www.worldlabs.ai/blog, 2025

    World Labs. Generating worlds.https://www.worldlabs.ai/blog, 2025

  50. [58]

    Exploring video quality assessment on user generated contents from aesthetic and technical perspectives

    Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou Hou, Annan Wang, Wenxiu Sun Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. InInternational Conference on Computer Vision, 2023

  51. [59]

    Worldmem: Long-term consistent world simulation with memory.arXiv preprint arXiv:2504.12369, 2025

    Zeqi Xiao, Yushi Lan, Yifan Zhou, Wenqi Ouyang, Shuai Yang, Yanhong Zeng, and Xingang Pan. Worldmem: Long-term consistent world simulation with memory.arXiv preprint arXiv:2504.12369, 2025

  52. [60]

    Gmflow: Learning optical flow via global matching

    Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. Gmflow: Learning optical flow via global matching. InComputer Vision and Pattern Recognition, pages 8121–8130, 2022

  53. [61]

    Playable game generation.arXiv preprint arXiv:2412.00887, 2024

    Mingyu Yang, Junyou Li, Zhongbin Fang, Sheng Chen, Yangbin Yu, Qiang Fu, Wei Yang, and Deheng Ye. Playable game generation.arXiv preprint arXiv:2412.00887, 2024

  54. [62]

    Learning interactive real-world simulators

    Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kaelbling, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. InInternational Conference on Learning Representations, 2024

  55. [63]

    Position: video as the new language for real-world decision making

    Sherry Yang, Jacob C Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, Andre Barreto, Pieter Abbeel, and Dale Schuurmans. Position: video as the new language for real-world decision making. InInternational Conference on Machine Learning, 2024

  56. [64]

    Direct-a-video: Customized video generation with user-directed camera movement and object motion

    Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. InACM SIGGRAPH, pages 1–12, 2024. 21

  57. [65]

    Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

  58. [66]

    Position: Interactive generative video as next-generation game engine.arXiv preprint arXiv:2503.17359, 2025

    Jiwen Yu, Yiran Qin, Haoxuan Che, Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Position: Interactive generative video as next-generation game engine.arXiv preprint arXiv:2503.17359, 2025

  59. [67]

    Gamefactory: Creating new games with generative interactive videos.arXiv preprint arXiv:2501.08325, 2025

    Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Gamefactory: Creating new games with generative interactive videos.arXiv preprint arXiv:2501.08325, 2025

  60. [68]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. InInternational Conference on Learning Representations, 2024

  61. [69]

    Hipa: enabling one-step text-to-image diffusion models via high-frequency- promoting adaptation.arXiv preprint arXiv:2311.18158, 2023

    Yifan Zhang and Bryan Hooi. Hipa: enabling one-step text-to-image diffusion models via high-frequency- promoting adaptation.arXiv preprint arXiv:2311.18158, 2023

  62. [70]

    Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition

    Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition. InAdvances in Neural Information Processing Systems, volume 35, pages 34077–34090, 2022

  63. [71]

    Deep long-tailed learning: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10795–10816, 2023

    Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10795–10816, 2023

  64. [72]

    Expanding small-scale datasets with guided imagination

    Yifan Zhang, Daquan Zhou, Bryan Hooi, Kai Wang, and Jiashi Feng. Expanding small-scale datasets with guided imagination. InAdvances in Neural Information Processing Systems, volume 36, pages 76558–76618, 2023

  65. [73]

    Memo: Memory-guided diffusion for expressive talking video generation

    Longtao Zheng, Yifan Zhang, Hanzhong Guo, Jiachun Pan, Zhenxiong Tan, Jiahao Lu, Chuanxin Tang, Bo An, and Shuicheng Yan. Memo: Memory-guided diffusion for expressive talking video generation. arXiv preprint arXiv:2412.04448, 2024

  66. [74]

    Is sora a world simulator? a comprehensive survey on general world models and beyond.arXiv preprint arXiv:2405.03520, 2024

    Zheng Zhu, Xiaofeng Wang, Wangbo Zhao, Chen Min, Nianchen Deng, Min Dou, Yuqi Wang, Botian Shi, Kai Wang, Chi Zhang, et al. Is sora a world simulator? a comprehensive survey on general world models and beyond.arXiv preprint arXiv:2405.03520, 2024. 22 (a) Camera↑ (b) Camera↓ (c...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.