REVIEW 3 major objections 7 minor 15 cited by
Matrix-Game: Interactive World Foundation Model
T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Matrix-Game claims that a 17B-parameter image-to-world diffusion model generates Minecraft videos that follow keyboard and mouse commands with 95% accuracy and that it outperforms the open-source baselines Oasis and MineWorld on every…
desk verdict A serious large-scale contribution, but the controllability numbers rest on an unvalidated IDM metric and need ablations, error bars, and artifact release before being trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing design is an action-conditioned latent diffusion transformer that generates video in a spatiotemporally compressed latent space built by a 3D causal VAE. The model concatenates the latent of the last five generated frames as motion context with a binary mask, applies a token-replacement trick inherited from prior image-to-video work for stable image-to-video generation, and injects control signals through a module that applies the group-operation trick to align action tokens with temporally compressed latents: keyboard actions enter through cross-attention, mouse movements through an MLP plus temporal self-attention. Training uses the rectified flow loss in two stages—unlabeled pretraining on 2,700 hours of filtered gameplay, then action-labeled training on about 1,200 hours—and autoregressive long-video generation is achieved by chaining segments with noise and classifier-free guidance on the motion context.
What would settle it
Take Table 2's action-controllability test and replace the IDM with an action estimator trained exclusively on synthetic Minecraft-like frames, then re-measure keyboard and mouse accuracy on the same generated clips; if accuracy drops substantially below 0.95 while the videos look unchanged, the reported control figures were inflated by IDM blind spots.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that an image-to-world latent diffusion transformer, scaled to 17B parameters and trained first without labels and then with action labels, can produce visually and temporally coherent Minecraft videos that accurately mirror the user's keyboard and mouse inputs. The model conditions only on a reference image and motion context, not on text, and employs an action-control module: discrete keyboard actions are injected through cross-attention, continuous mouse angles through temporal self-attention, with classifier-free guidance applied to actions, motion frames, and the reference image. Evaluated by the paper's GameWorld Score, the model reports 0.95 keyboard accuracy and 0.95 mouse accuracy, versus 0.86 and 0.64 for the best prior open-source baseline (MineWorld), and the authors report higher object and scenario consistency.
Load-bearing premise
The reported control accuracy assumes that the Inverse Dynamics Model trained on 1,962 hours of real Minecraft gameplay gives valid action labels when applied to generated video; if the generated frames exploit statistical regularities the IDM has memorized, the 95% keyboard and mouse accuracies would overstate true control.
Editorial extensions
If this is right
- If the reported accuracies hold, interactive video generation from a single image becomes a practical way to explore and steer open-ended game worlds without a hand-written game engine.
- The two-stage training recipe—unlabeled world pretraining followed by action-labeled fine-tuning—becomes a default template for building controllable world models in other games and simulated environments.
- The GameWorld Score benchmark supplies a standardized set of eight measurements that future Minecraft world models can be compared against.
- Autoregressive chaining of action-conditioned clips with a five-frame motion context extends controllability to arbitrarily long videos while keeping temporal coherence across segment boundaries.
Reading between the lines
- The IDM-based controllability metric may overestimate true control: if the IDM was trained on the same visual style of Minecraft footage, it could be exploiting appearance-level regularities rather than verifying that the scene actually responds to the action semantics; a human-in-the-loop test where users issue commands to a live model would be a stronger check.
- The method's reliance on Minecraft-specific unlabeled data and IDM labels means the generalization claim to other games (such as racing or combat simulators) is, for now, a roadmap rather than a demonstrated result.
- A testable extension would be to measure whether the same two-stage pipeline with a separate, non-Minecraft dataset (for example, a physics sandbox) yields similar control accuracy, which would separate the contribution of the architecture from the contribution of the Minecraft-specific data.
- If GameWorld Score becomes standard, its scenario-consistency test—which pairs symmetric camera motions and requires the scene to reappear—could serve as a useful proxy for whether a world model truly reasons about a persistent 3D scene rather than a bag of frames.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Matrix-Game, a 17B-parameter latent diffusion model for interactive Minecraft world generation, conditioned on a reference image, motion context, and keyboard/mouse actions. It introduces a two-stage training pipeline and the Matrix-Game-MC dataset (about 2,700 hours of unlabeled video and 1,000–1,200 hours of action-labeled video), as well as GameWorld Score, an eight-metric benchmark covering visual quality, temporal quality, action controllability, and physical consistency. The authors report that Matrix-Game outperforms Oasis and MineWorld on all benchmark dimensions, with keyboard and mouse accuracies of 0.95, and that double-blind human evaluation confirms its superiority. They plan to release model weights and benchmark code.
Significance. If the claims hold, this is a substantial open-source contribution to interactive world modeling, combining a large-scale dataset, a two-stage training scheme, and a multi-dimensional evaluation suite. The paper is honest about failure cases and remaining limitations, which is a strength. However, the central quantitative evidence for controllability and the headline comparison rests on a newly proposed benchmark whose action-controllability component may be confounded with the action-labeling pipeline used to train the data, and the paper does not report error bars, evaluation-set details, or validation of the IDM on generated video. The human study supports the direction but is under-reported. Because the method and dataset are plausible and the issues are addressable through additional analysis, the manuscript merits major revision rather than rejection.
major comments (3)
- [§5.3, Tables 2–3] The action-controllability pillar labels generated videos with the same class of Inverse Dynamics Model (IDM) that is embedded elsewhere in the data pipeline: Section 3.1 uses an IDM for menu-state and camera-movement filtering, and Section 3.2 constructs action labels via VPT agents, which rely on IDM-style action inference. The benchmark's validity therefore depends on the IDM's precision on synthetic video, which the paper does not report. I ask for (i) a statement of whether the evaluation IDM is the same checkpoint as the one used in data creation; (ii) a calibration experiment reporting IDM accuracy on generated videos with known ground-truth actions, split by scenario; and (iii) a comparison of IDM-based controllability scores with human judgments of action following on the same clips. This is load-bearing because Tables 2 and 3 are the primary evidence for the central claim of precise control; the failure cases in Section 6.5 show that physically implausible generations can still receive high IDM-based accuracy, confirming that the metric may reward style mimicry rather than true action following.
- [§6.1, Table 2 and Figure 8] The headline comparisons are reported as point estimates without error bars, number of evaluation videos, seeds, or significance tests, so it is impossible to judge whether the margins over Oasis and MineWorld are meaningful. The evaluation set should be specified: how many reference images, action sequences, clip lengths, and generation seeds per model, and whether the same reference images and action sequences were used for all models. The human study in Figure 8 reports only aggregate win rates; please provide the number of annotators, number of comparisons per dimension, instructions, inter-annotator agreement, and significance of the win-rate differences. Without this information, the claim that Matrix-Game 'consistently outperforms' the baselines is not fully supported.
- [§3.2 and §4.2] The labeled dataset size is given inconsistently: Section 3.2 reports 'over 1,026 hours of video clips for 33-frame training' and separately 'more than 1,200 hours of video for 65-frame training,' whereas Section 4.2 Stage 2 states the model is 'trained on 1,200 hours of action-labeled 720p 33-frame video clips' and then later uses 'approximately 1,200 hours of 720p 65-frame video clips.' Please clarify which dataset was used in each training stage and reconcile the 1,026-hour and 1,200-hour numbers; if two different labeled sets were used, state the order and how they relate.
minor comments (7)
- [§5.3] The mouse-direction evaluation uses a 'pre-defined threshold' for detecting rotational change, but the threshold value is never given; please specify it and describe how continuous predicted yaw and pitch are mapped to the eight directional categories.
- [§5.3] The IDM used for evaluation is described only as 'trained on 1,962 hours of Minecraft gameplay' with no citation or checkpoint identifier; please state whether it is the VPT IDM of [4], a MineWorld [18] release, or a new model, and provide the source.
- [§6.1, Figure 8] Calling the human study 'double-blind' is imprecise: the annotators are blind to method identity, but the selection of videos and their pairing are not described; please specify who selected the videos, how many were used per condition, and the presentation order and randomization.
- [§2.3] In the sentence listing prior game-video-generation works, reference [41] appears twice; please remove the duplicate.
- [§6.2, Table 3] The text says the model attains 'over 88% accuracy on all actions,' but the jump accuracy in Table 3 is exactly 0.88; either rephrase to 'at least 88%' or correct the number.
- [§4.1, §5.3, §7] The paper states in Section 4.1 that mouse movements are continuous scalar values, but Section 5.3 evaluates only eight directions, and Section 7 lists 'mouse control with a broader and more continuous value range' as future work; please clarify whether the trained model already supports continuous mouse input and why the benchmark only tests discrete directions.
- [Abstract and §5] The paper announces that code and weights will be open-sourced, but provides no link, license, or release timeline; please add concrete availability information or state that the release is pending final approval.
Circularity Check
No circularity: the claimed controllability gains are post-hoc measurements with an external IDM and independent human evaluation, not fitted predictions.
full rationale
I walked the paper's derivation chain and found no step in which a prediction or claimed result reduces by construction to its inputs. Matrix-Game is trained with a rectified-flow loss on unlabeled and action-labeled video (Sections 3–4); the keyboard/mouse accuracies in Tables 2–3 are computed after training by GameWorld Score, whose action-controllability pillar uses an Inverse Dynamics Model adopted from MineWorld/VPT (Section 5.3) to label generated videos. The IDM is not trained by the authors on Matrix-Game outputs, and no parameter of Matrix-Game is fitted to the benchmark, so the 0.95/0.95 numbers are measurements rather than fit-derived predictions. The IDM is also used in data curation (menu-state and camera-movement filtering, Section 3.1), but that only weakens the metric's independence; it does not make the model's control accuracy true by definition. The human double-blind evaluation (Figure 8) and the external baselines Oasis and MineWorld provide independent support for the ranking. Self-citations appear only in routine methodological contexts (e.g., references [8, 34, 69–73] for augmentation and long-tailed training) and are not load-bearing for the central claim. The admitted failure cases in Section 6.5 (e.g., walking through leaves) qualify the physical-consistency claim but are limitations, not circularity. Therefore no circular step is identified.
Assumptions & free parameters
free parameters (3)
- per-frame yaw/pitch rotation cap in labeled data creation =
15 degrees
- IDM mouse-movement detection threshold =
not specified
- CFG scales and probabilities =
CFG 6 for actions/motion/reference; action dropout 0.1; motion dropout 0.25
assumptions (4)
- domain assumption IDM trained on 1,962 hours of Minecraft gameplay is a valid action oracle for generated video
- domain assumption Unreal Engine procedural videos transfer to Minecraft dynamics
- domain assumption Pretrained HunyuanVideo weights provide a useful visual prior after replacing the text branch
- domain assumption Laion aesthetic, MUSIQ, CLIP, AMT, and DROID-SLAM are valid no-reference metrics for game video
Cite this review
Pith. "Pith review of Matrix-Game: Interactive World Foundation Model." pith.science (2026). https://pith.science/paper/V2T64JEV
@misc{pith2026250618701,
author = {Pith},
title = {Pith review of: Matrix-Game: Interactive World Foundation Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/V2T64JEV}},
note = {Machine review of arXiv:2506.18701}
}
read the original abstract
We introduce Matrix-Game, an interactive world foundation model for controllable game world generation. Matrix-Game is trained using a two-stage pipeline that first performs large-scale unlabeled pretraining for environment understanding, followed by action-labeled training for interactive video generation. To support this, we curate Matrix-Game-MC, a comprehensive Minecraft dataset comprising over 2,700 hours of unlabeled gameplay video clips and over 1,000 hours of high-quality labeled clips with fine-grained keyboard and mouse action annotations. Our model adopts a controllable image-to-world generation paradigm, conditioned on a reference image, motion context, and user actions. With over 17 billion parameters, Matrix-Game enables precise control over character actions and camera movements, while maintaining high visual quality and temporal coherence. To evaluate performance, we develop GameWorld Score, a unified benchmark measuring visual quality, temporal quality, action controllability, and physical rule understanding for Minecraft world generation. Extensive experiments show that Matrix-Game consistently outperforms prior open-source Minecraft world models (including Oasis and MineWorld) across all metrics, with particularly strong gains in controllability and physical consistency. Double-blind human evaluations further confirm the superiority of Matrix-Game, highlighting its ability to generate perceptually realistic and precisely controllable videos across diverse game scenarios. To facilitate future research on interactive image-to-world generation, we will open-source the Matrix-Game model weights and the GameWorld Score benchmark at https://github.com/SkyworkAI/Matrix-Game.
Figures
Figures from the paper (12 more)
Forward citations
Cited by 15 Pith papers
-
Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models
Per-frame natural-language action prompts enable simultaneous multi-entity control and cross-entity action transfer in interactive video world models, outperforming discrete action-index interfaces.
-
ActionParty: Multi-Subject Action Binding in Generative Video Games
ActionParty binds discrete actions to individual subjects in a single generated video by jointly modeling subject state tokens and video latents, controlling up to seven players across 46 Melting Pot games.
-
RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation
Step Forcing trains a few-step autoregressive video world model so RoboWorld closed-loop rollouts plus a task-progress VLM judge recover real-world policy rankings at r=0.989 and ρ=0.970.
-
Alaya-EVOKE: From Linear-Scaling Supervision to Endless World
Evoke combines an external camera-indexed geometry memory with a long-horizon teacher to produce a three-step world model that generates hour-scale video with bounded per-step cost.
-
WorldCycle: Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models
WorldCycle post-trains interactive video world models with reinforcement learning rewards for spatial closure and temporal consistency on reversible action cycles, reducing long-horizon drift and improving composite-a...
-
Wonder: Video World Model Done Better
Wonder generates minute-scale, real-time camera-controllable video worlds from a single image or video at 16 FPS, using a rendered coordinate-field control signal, sparse full-fidelity memory, and stage-specialized di...
-
DynaVieW: Schema-Guided World Modeling for Understanding Hierarchical Visual Dynamics
Schema-guided interleaved state-transition pretraining with selective attention and reweighted loss improves hierarchical visual dynamics modeling for narrative generation and world simulation.
-
Worldscape-MoE: A Unified Mixture-of-Experts World Model for Scalable Heterogeneous Action Control
A DiT Mixture-of-Experts world model jointly learns locomotion, dual-arm manipulation, and egocentric hand control, with shared experts for world dynamics and progressive expert expansion for new modalities.
-
WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL
WoVR shows that reinforcement learning can improve VLA robot policies through imagined rollouts in a video world model, reporting +29.3 points on LIBERO and +30.0 points on real Franka tasks.
-
CustomX: Unified Character, Action, and Scene Customization in Video World Models
AniX generates controllable videos of a user-supplied character performing typed actions inside a user-supplied 3D scene by fine-tuning a pre-trained video generator on small locomotion datasets.
-
End-to-End Training for Autoregressive Video Diffusion via Self-Resampling
Resampling Forcing trains autoregressive video diffusion models on self-resampled degraded histories with a causal mask, achieving stable long-horizon generation without a teacher or discriminator.
-
Quo Vadis, World Modeling?
An agent-centric reframing of world modeling, replacing physical state prediction with 'information transitions' organized into six proxy functions and three empowerment levels.
-
From Pixels to States: Rethinking Interactive World Models as Game Engines
Interactive world models are reorganized around the game-engine action-state-observation loop, and a 90-hour Black Myth: Wukong dataset with frame-aligned actions, ground-truth states, and observations is introduced.
-
Resilience Revisited: A Multidimensional Framework Derived from Realistic Attack Scenarios
A new MDRI framework decomposes power system resilience into five dimensions with a calibrated multiplicative term, showing multi-vector attacks cause 5.6 times more degradation than linear sums plus 60.6% from coupling.
-
OpenWorldLib: A Unified Codebase and Definition of Advanced World Models
OpenWorldLib offers a standardized codebase and definition for world models that combine perception, interaction, and memory to understand and predict the world.
Reference graph
Works this paper leans on
-
[1]
Cosmos: world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575, 2025
Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos: world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575, 2025
arXiv 2025
-
[2]
Diffusion for world modeling: Visual details matter in atari
Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos J Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. InAdvances in Neural Information Processing Systems, volume 37, pages 58757–58791, 2024
work page 2024
-
[3]
Character region awareness for text detection
Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee. Character region awareness for text detection. InComputer Vision and Pattern Recognition, pages 9365–9374, 2019
work page 2019
-
[4]
Video pretraining (vpt): Learning to act by watching unlabeled online videos
Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. InAdvances in Neural Information Processing Systems, volume 35, pages 24639–24654, 2022
work page 2022
-
[5]
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023
arXiv 2023
-
[6]
Genie: Generative interactive environments
Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. InInternational Conference on Machine Learning, 2024
2024
-
[7]
Gamegen-x: Interactive open-world game video generation.arXiv preprint arXiv:2411.00769, 2024
Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamegen-x: Interactive open-world game video generation.arXiv preprint arXiv:2411.00769, 2024
arXiv 2024
-
[8]
One-dm: One-shot diffusion mimicker for handwritten text generation
Gang Dai, Yifan Zhang, Quhui Ke, Qiangya Guo, and Shuangping Huang. One-dm: One-shot diffusion mimicker for handwritten text generation. InEuropean Conference on Computer Vision, pages 410–427. Springer, 2024
work page 2024
Show all 74 references
-
[9]
Oasis: A universe in a transformer
Decart. Oasis: A universe in a transformer. 2024
2024
-
[10]
Worldscore: A unified evaluation benchmark for world generation.arXiv preprint arXiv:2504.00983, 2025
Haoyi Duan, Hong-Xing Yu, Sirui Chen, Li Fei-Fei, and Jiajun Wu. Worldscore: A unified evaluation benchmark for world generation.arXiv preprint arXiv:2504.00983, 2025
2025
-
[11]
Scaling rectified flow transformers for high-resolution image synthesis
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. InInternational Conference on Machine Learning, 2024
2024
-
[12]
Minedojo: Building open-ended embodied agents with internet-scale knowledge
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. InAdvances in Neural Information Processing Systems, volume 35...
2022
-
[13]
Perceptual quality assessment of smartphone photography
Yuming Fang, Hanwei Zhu, Yan Zeng, Kede Ma, and Zhou Wang. Perceptual quality assessment of smartphone photography. InComputer Vision and Pattern Recognition, 2020
2020
-
[14]
The matrix: Infinite-horizon world generation with real-time moving control
Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, and Hongyang Zhang. The matrix: Infinite-horizon world generation with real-time moving control. arXiv preprint arXiv:2412.03568, 2024
2024 arXiv
-
[15]
FFmpeg: A complete, cross-platform solution to record, convert and stream audio and video.https://ffmpeg.org/, 2024
FFmpeg Team. FFmpeg: A complete, cross-platform solution to record, convert and stream audio and video.https://ffmpeg.org/, 2024
2024
-
[16]
Vista: A generalizable driving world model with high fidelity and versatile controllability
Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. arXiv preprint arXiv:2405.17398, 2024
2024 arXiv
-
[17]
Long-context autoregressive video modeling with next-frame prediction.arXiv preprint arXiv:2503.19325, 2025
Yuchao Gu, weijia Mao, and Mike Zheng Shou. Long-context autoregressive video modeling with next-frame prediction.arXiv preprint arXiv:2503.19325, 2025. 19
2025 arXiv
-
[18]
Mineworld: a real-time and open-source interactive world model on minecraft.arXiv preprint arXiv:2504.08388, 2025
Junliang Guo, Yang Ye, Tianyu He, Haoyu Wu, Yushu Jiang, Tim Pearce, and Jiang Bian. Mineworld: a real-time and open-source interactive world model on minecraft.arXiv preprint arXiv:2504.08388, 2025
2025 arXiv
-
[19]
The essential role of causality in foundation world models for embodied ai.arXiv preprint arXiv:2402.06665, 2024
Tarun Gupta, Wenbo Gong, Chao Ma, Nick Pawlowski, Agrin Hilmkil, Meyer Scetbon, Marc Rigter, Ade Famoti, Ashley Juan Llorens, Jianfeng Gao, et al. The essential role of causality in foundation world models for embodied ai.arXiv preprint arXiv:2402.06665, 2024
2024 arXiv
-
[20]
World models.arXiv preprint arXiv:1803.10122, 2018
David Ha and Jürgen Schmidhuber. World models.arXiv preprint arXiv:1803.10122, 2018
2018 arXiv
-
[21]
PhD thesis, University of Toronto (Canada), 2024
Danijar Hafner.Embodied Intelligence Through World Models. PhD thesis, University of Toronto (Canada), 2024
2024
-
[22]
Cameractrl: Enabling camera control for text-to-video generation.arXiv preprint arXiv:2404.02101, 2024
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation.arXiv preprint arXiv:2404.02101, 2024
2024 arXiv
-
[23]
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. InAdvances in Neural Information Processing Systems, volume 33, pages 6840–6851, 2020
2020
-
[24]
Gaia-1: A generative world model for autonomous driving.arXiv preprint arXiv:2309.17080, 2023
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving.arXiv preprint arXiv:2309.17080, 2023
2023 arXiv
-
[25]
Vbench: Comprehensive benchmark suite for video generative models
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. InComputer Vision and Pattern Recognition, pages 21807–21818, 2024
2024
-
[26]
MUSIQ: multi-scale image quality transformer.arXiv preprint arXiv:2108.05997, 2021
Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. MUSIQ: multi-scale image quality transformer.arXiv preprint arXiv:2108.05997, 2021
2021 arXiv
-
[27]
Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024
2024 arXiv
-
[28]
Auto-encoding variational bayes, 2013
Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013
2013
-
[29]
Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024
2024 arXiv
-
[30]
Flux.https://github.com/black-forest-labs/flux, 2024
Black Forest Labs. Flux.https://github.com/black-forest-labs/flux, 2024
2024
-
[31]
aesthetic-predictor.https://github.com/LAION-AI/aesthetic-predictor, 2022
LAION-AI. aesthetic-predictor.https://github.com/LAION-AI/aesthetic-predictor, 2022
2022
-
[32]
Amt: All-pairs multi-field transforms for efficient frame interpolation
Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. Amt: All-pairs multi-field transforms for efficient frame interpolation. InComputer Vision and Pattern Recognition, 2023
2023
-
[33]
Open-sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024
Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024
2024 arXiv
-
[34]
Drivegen: Generalized and robust 3d detection in driving via controllable text-to-image diffusion generation
Hongbin Lin, Zilu Guo, Yifan Zhang, Shuaicheng Niu, Yafeng Li, Ruimao Zhang, Shuguang Cui, and Zhen Li. Drivegen: Generalized and robust 3d detection in driving via controllable text-to-image diffusion generation. InComputer Vision and Pattern Recognition, 2025
2025
-
[35]
World model on million-length video and language with ringattention.arXiv e-prints, pages arXiv–2402, 2024
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention.arXiv e-prints, pages arXiv–2402, 2024
2024
-
[36]
Flow straight and fast: Learning to generate and transfer data with rectified flow
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. InInternational Conference on Learning Representations, 2023
2023
-
[37]
A survey on vision-language-action models for embodied ai.arXiv preprint arXiv:2405.14093, 2024
Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. A survey on vision-language-action models for embodied ai.arXiv preprint arXiv:2405.14093, 2024
2024 arXiv
-
[38]
MineRL: A large-scale dataset of minecraft demonstrations
MineRL Project. MineRL: A large-scale dataset of minecraft demonstrations. https://minerl. readthedocs.io/en/latest/, 2020
2020
-
[39]
Conditional image-to- video generation with latent flow diffusion models
Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image-to- video generation with latent flow diffusion models. InComputer Vision and Pattern Recognition, pages 18444–18455, 2023
2023
-
[40]
Sora: Video generation models as world simulators
OpenAI. Sora: Video generation models as world simulators. https://openai.com/index/ video-generation-models-as-world-simulators/, 2024
2024
-
[41]
Genie 2: A large-scale foundation world model.URL: https://deepmind
J Parker-Holder, P Ball, J Bruce, V Dasagi, K Holsheimer, C Kaplanis, A Moufarek, G Scully, J Shar, J Shi, et al. Genie 2: A large-scale foundation world model.URL: https://deepmind. google/discover/blog/genie- 2-a-large-scale-foundation-world-model, 2024
2024
-
[42]
Scalable diffusion models with transformers
William Peebles and Saining Xie. Scalable diffusion models with transformers. InInternational Conference on Computer Vision, pages 4195–4205, 2023. 20
2023
-
[43]
SDXL: Improving latent diffusion models for high-resolution image synthesis
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, 2023
2023
-
[44]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning, 2021
2021
-
[45]
High-resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InComputer Vision and Pattern Recognition, pages 10684– 10695, 2022
2022
-
[46]
A benchmark of facial recognition pipelines and co-usability perfor- mances of modules.Journal of Information Technologies, 17(2):95–107, 2024
Sefik Serengil and Alper Ozpinar. A benchmark of facial recognition pipelines and co-usability perfor- mances of modules.Journal of Information Technologies, 17(2):95–107, 2024
2024
-
[47]
Deep unsupervised learning using nonequilibrium thermodynamics
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InInternational Conference on Machine Learning, pages 2256–2265. PMLR, 2015
2015
-
[48]
Score-based generative modeling through stochastic differential equations
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. InInternational Conference on Learning Representations, 2021
2021
-
[49]
Transnet v2: An effective deep network architecture for fast shot transition detection
Tomás Soucek and Jakub Lokoc. Transnet v2: An effective deep network architecture for fast shot transition detection. InACM International Conference on Multimedia, pages 11218–11221, 2024
2024
-
[50]
Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras
Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. In Advances in Neural Information Processing Systems, volume 34, pages 16558–16569, 2021
2021
-
[51]
Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions
Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. InEuropean Conference on Computer Vision, pages 244–260. Springer, 2024
2024
-
[52]
Diffusion models are real-time game engines.arXiv preprint arXiv:2408.14837, 2024
Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines.arXiv preprint arXiv:2408.14837, 2024
2024 arXiv
-
[53]
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems, volume 30, 2017
2017
-
[54]
Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025
Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025
2025 arXiv
-
[55]
Drivedreamer: Towards real-world-drive world models for autonomous driving
Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world-drive world models for autonomous driving. InEuropean Conference on Computer Vision, pages 55–72, 2024
2024
-
[56]
Motionctrl: A unified and flexible motion controller for video generation
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. InACM SIGGRAPH, pages 1–11, 2024
2024
-
[57]
Generating worlds.https://www.worldlabs.ai/blog, 2025
World Labs. Generating worlds.https://www.worldlabs.ai/blog, 2025
2025
-
[58]
Exploring video quality assessment on user generated contents from aesthetic and technical perspectives
Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou Hou, Annan Wang, Wenxiu Sun Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. InInternational Conference on Computer Vision, 2023
2023
-
[59]
Worldmem: Long-term consistent world simulation with memory.arXiv preprint arXiv:2504.12369, 2025
Zeqi Xiao, Yushi Lan, Yifan Zhou, Wenqi Ouyang, Shuai Yang, Yanhong Zeng, and Xingang Pan. Worldmem: Long-term consistent world simulation with memory.arXiv preprint arXiv:2504.12369, 2025
2025
-
[60]
Gmflow: Learning optical flow via global matching
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. Gmflow: Learning optical flow via global matching. InComputer Vision and Pattern Recognition, pages 8121–8130, 2022
2022
-
[61]
Playable game generation.arXiv preprint arXiv:2412.00887, 2024
Mingyu Yang, Junyou Li, Zhongbin Fang, Sheng Chen, Yangbin Yu, Qiang Fu, Wei Yang, and Deheng Ye. Playable game generation.arXiv preprint arXiv:2412.00887, 2024
2024 arXiv
-
[62]
Learning interactive real-world simulators
Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kaelbling, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. InInternational Conference on Learning Representations, 2024
2024
-
[63]
Position: video as the new language for real-world decision making
Sherry Yang, Jacob C Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, Andre Barreto, Pieter Abbeel, and Dale Schuurmans. Position: video as the new language for real-world decision making. InInternational Conference on Machine Learning, 2024
2024
-
[64]
Direct-a-video: Customized video generation with user-directed camera movement and object motion
Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. InACM SIGGRAPH, pages 1–12, 2024. 21
2024
-
[65]
Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024
2024 arXiv
-
[66]
Position: Interactive generative video as next-generation game engine.arXiv preprint arXiv:2503.17359, 2025
Jiwen Yu, Yiran Qin, Haoxuan Che, Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Position: Interactive generative video as next-generation game engine.arXiv preprint arXiv:2503.17359, 2025
2025 arXiv
-
[67]
Gamefactory: Creating new games with generative interactive videos.arXiv preprint arXiv:2501.08325, 2025
Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Gamefactory: Creating new games with generative interactive videos.arXiv preprint arXiv:2501.08325, 2025
2025
-
[68]
Language model beats diffusion–tokenizer is key to visual generation
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. InInternational Conference on Learning Representations, 2024
2024
-
[69]
Hipa: enabling one-step text-to-image diffusion models via high-frequency- promoting adaptation.arXiv preprint arXiv:2311.18158, 2023
Yifan Zhang and Bryan Hooi. Hipa: enabling one-step text-to-image diffusion models via high-frequency- promoting adaptation.arXiv preprint arXiv:2311.18158, 2023
2023 arXiv
-
[70]
Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition
Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition. InAdvances in Neural Information Processing Systems, volume 35, pages 34077–34090, 2022
2022
-
[71]
Deep long-tailed learning: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10795–10816, 2023
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10795–10816, 2023
2023
-
[72]
Expanding small-scale datasets with guided imagination
Yifan Zhang, Daquan Zhou, Bryan Hooi, Kai Wang, and Jiashi Feng. Expanding small-scale datasets with guided imagination. InAdvances in Neural Information Processing Systems, volume 36, pages 76558–76618, 2023
2023
-
[73]
Memo: Memory-guided diffusion for expressive talking video generation
Longtao Zheng, Yifan Zhang, Hanzhong Guo, Jiachun Pan, Zhenxiong Tan, Jiahao Lu, Chuanxin Tang, Bo An, and Shuicheng Yan. Memo: Memory-guided diffusion for expressive talking video generation. arXiv preprint arXiv:2412.04448, 2024
2024 arXiv
-
[74]
Is sora a world simulator? a comprehensive survey on general world models and beyond.arXiv preprint arXiv:2405.03520, 2024
Zheng Zhu, Xiaofeng Wang, Wangbo Zhao, Chen Min, Nianchen Deng, Min Dou, Yuqi Wang, Botian Shi, Kai Wang, Chi Zhang, et al. Is sora a world simulator? a comprehensive survey on general world models and beyond.arXiv preprint arXiv:2405.03520, 2024. 22 (a) Camera↑ (b) Camera↓ (c...
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.