REVIEW 27 cited by
Pandora: Towards General World Model with Natural Language Actions and Video States
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
World models simulate future states of the world in response to different actions. They facilitate interactive content creation and provides a foundation for grounded, long-horizon reasoning. Current foundation models do not fully meet the capabilities of general world models: large language models (LLMs) are constrained by their reliance on language modality and their limited understanding of the physical world, while video models lack interactive action control over the world simulations. This paper makes a step towards building a general world model by introducing Pandora, a hybrid autoregressive-diffusion model that simulates world states by generating videos and allows real-time control with free-text actions. Pandora achieves domain generality, video consistency, and controllability through large-scale pretraining and instruction tuning. Crucially, Pandora bypasses the cost of training-from-scratch by integrating a pretrained LLM (7B) and a pretrained video model, requiring only additional lightweight finetuning. We illustrate extensive outputs by Pandora across diverse domains (indoor/outdoor, natural/urban, human/robot, 2D/3D, etc.). The results indicate great potential of building stronger general world models with larger-scale training.
Forward citations
Cited by 27 Pith papers
-
MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
None of ten tested video-generation models reliably remembers objects after occlusion in dynamic scenes; static-camera videos inflate consistency scores.
-
Video Creation by Demonstration
A self-supervised diffusion approach, δ-Diffusion, transfers action concepts from a demonstration video to a new context image using appearance-bottlenecked action latents.
-
PhiZero: A World Model Built Around Physical Language
A self-supervised discrete physical-language bottleneck plus a VLM reasoner lets a world model predict state transitions before rendering video, improving physical coherence and enabling zero-shot motion transfer.
-
GS-Agent: Creating 4D Physical Worlds With Generative Simulation
Three LLM agents write physics-engine code from text, review rendered frames, and correct errors, turning prompts into physically simulated 4D worlds with camera control.
-
RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation
A tri-branch diffusion model co-generates RGB, depth, and optical flow from a single RGB-D image, and an inverse dynamics head on its internal latents achieves state-of-the-art bimanual manipulation success rates.
-
Flow Equivariant World Models: Memory for Partially Observed Dynamic Environments
Flow equivariant world models use a latent memory that shifts with the agent and with inferred object motion, giving stable long-horizon prediction under partial observability.
-
A Comprehensive Survey on World Models for Embodied AI
A unified three-axis taxonomy — functionality, temporal modeling, spatial representation — organizes the world-model literature for embodied AI.
-
LoViC: Efficient Long Video Generation with Context Compression
LoViC uses FlexFormer, a single-query-token Q-Former with interpolated rotary positional encoding, to compress long video-text context for efficient long-video generation.
-
GenWorld: Towards Detecting AI-generated Real-world Simulation Videos
GenWorld is a 100k real-world-simulation video forgery benchmark, and SpannDetector uses multi-view 3D consistency to detect AI-generated videos, especially world-model outputs that fool existing detectors.
-
AnimeShooter: A Multi-Shot Animation Dataset for Reference-Guided Video Generation
AnimeShooter provides hierarchical story and shot annotations plus reference images for 148K one-minute animation stories, and AnimeShooterGen trained on it shows improved cross-shot consistency.
-
Playing with Transformer at 30+ FPS via Next-Frame Diffusion
Next-Frame Diffusion combines block-wise causal attention, consistency distillation, and action-based speculative sampling to generate action-conditioned Minecraft video at over 30 FPS on an A100 with a 310M parameter model.
-
Humanoid World Models: Open World Foundation Models for Humanoid Robotics
Masked-transformers trained on humanoid video forecast future frames with better FID than flow-matching models, and parameter sharing cut model size 33-53% with minimal quality loss.
-
Long-Context State-Space Video World Models
A hybrid state-space and local-attention architecture gives autoregressive video diffusion models long-term spatial memory with constant per-frame inference cost, demonstrated on Maze and Minecraft.
-
TesserAct: Learning 4D Embodied World Models
A 4D embodied world model that generates RGB-depth-normal videos from an image and instruction, reconstructs the scene as point clouds, and uses those point clouds to train better robot manipulation policies.
-
Learning Human Skill Generators at Key-Step Levels
A benchmark and framework for generating key-step video clips of human skills from one initial image and a skill description.
-
Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile
A three-stage pipeline combining sparse 'tile' attention with multi-step consistency distillation makes Open-Sora-Plan video generation up to 7.8x faster while keeping the aggregate VBench final score within 1%.
-
EgoAgent: A Joint Predictive Agent Model in Egocentric Worlds
EgoAgent is a single transformer trained from scratch on egocentric video and 3D poses that jointly learns image representations, latent next-state prediction, and 3D human motion prediction, with ablations showing mu...
-
Generative Physical AI in Vision: A Survey
A structured review that categorizes physics-aware generative models in vision into explicit-simulation and implicit-learning families and proposes six integration paradigms.
-
Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generation
A new story-completion benchmark, StoryEval, shows that 11 current text-to-video models complete fewer than half of the consecutive events in short story prompts.
-
GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control
GEM generates controllable future RGB and depth ego-vision frames, conditioned on ego-trajectories, sparse object tokens, and human poses, across driving, egocentric, and drone domains.
-
The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control
A 2.7B parameter diffusion model trained on game and internet footage generates control-responsive 720p video streams, but the paper's 'infinite, real-time, zero-shot' claims are not backed by public benchmarks or rel...
-
From Pixels to States: Rethinking Interactive World Models as Game Engines
Interactive world models are reorganized around the game-engine action-state-observation loop, and a 90-hour Black Myth: Wukong dataset with frame-aligned actions, ground-truth states, and observations is introduced.
-
Pre-Trained Video Generative Models as World Simulators
A lightweight action-conditioning module and a motion-reinforced loss convert pre-trained video generators into action-following world simulators that also speed up model-based reinforcement learning.
-
ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restoration
ReconDreamer fine-tunes a driving world model as an online restorer and progressively expands novel-trajectory training data, reporting first-time effective rendering of multi-lane shifts in driving scenes.
-
GigaWorld-Policy-0.5: A Faster and Stronger WAM Empowered by AutoResearch
GigaWorld-Policy-0.5 uses a Mixture-of-Transformers action-expert split and mixed world-model pretraining to reach 85 ms action-only inference with claimed success-rate gains.
-
From World Action Models to Embodied Brains: A Roadmap for Open-World Physical Intelligence
Physical intelligence needs an embodied brain that reasons over interventions and emits capability requests, grounded by a physical harness and shared experience contracts rather than direct actuator policies.
-
Critique of World Model
The paper argues world models should simulate actionable possibilities and proposes GLP, a hierarchical generative architecture that closes the loop with observation reconstruction, but provides no experiments.
Discussion (0). Continue with ORCID to comment.