Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

By binding each action to a latent subject token, a video-diffusion world model controls up to seven players in one generated scene.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 00:51 UTC pith:JAFMZ3H4

load-bearing objection ActionParty makes a real contribution to multi-agent world models, but the headline numbers rest on metrics that need validation on generated frames. the 4 major comments →

arxiv 2604.02330 v2 pith:JAFMZ3H4 submitted 2026-04-02 cs.CV cs.AIcs.LG

ActionParty: Multi-Subject Action Binding in Generative Video Games

Alexander Pondaven , Ziyi Wu , Igor Gilitschenski , Philip Torr , Sergey Tulyakov , Fabio Pizzati , Aliaksandr Siarohin This is my paper
classification cs.CV cs.AIcs.LG
keywords video world modelsaction bindingmulti-agent controlsubject state tokensattention maskingrotary position embeddingsgenerative game enginesdiffusion transformers
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ActionParty tackles the action-binding problem in generative video games: when several players appear in one scene, a video-diffusion model often applies an action to the wrong character or drops a player entirely. The paper argues this happens because actions have no explicit anchor to subjects, and proposes to add a persistent latent token per subject that records the subject's 2D position and is denoised together with the video frames. Two attention masks and a rotary-position bias then enforce that each action updates only its own subject token and that each token reads only from the pixels near its current location. On a benchmark of 46 multi-agent games, the resulting model is the first to control up to seven players simultaneously, with action-following accuracy around 78 percent versus 16 percent for the best text-only baseline. A sympathetic reader would care because multi-agent simulation is exactly what current single-agent world models cannot do, and the paper shows a concrete architectural fix rather than a prompt engineering workaround.

Core claim

The central claim is that action binding can be solved by making subject identity explicit in the latent space. Instead of asking the model to infer 'who does what' from pixels and text, ActionParty adds one state token per player, defined as its current 2D coordinates, and jointly denoises those tokens with the video latent. A cross-attention mask lets each subject token attend only to its own action embedding, so updates are per-subject; a self-attention mask lets video tokens read all subject states while blocking subject-to-subject mixing; and a 3D rotary position embedding biases each subject token to attend to video tokens at its previous spatial position. The paper reports that this r

What carries the argument

The central object is the subject state token: a latent variable holding a player's 2D coordinates, concatenated with the video tokens and denoised jointly. Its work is carried by two masked-attention mechanisms and a spatial bias. In cross-attention, a binary mask restricts each subject token to attend only to its own action embedding, enforcing strict subject-action binding. In self-attention, a mask blocks subject-to-subject attention while allowing video tokens to see all subjects, and a 3D Rotary Position Embedding (RoPE) rotates subject tokens so they attend preferentially to video tokens at the subject's previous position. Together these turn 'render the frame' and 'update the player'

Load-bearing premise

The method assumes that the first frame and the exact 2D starting position of every subject are given, and that a subject's state is fully captured by its location; if those positions are wrong, unavailable, or two subjects occupy the same spot, action binding collapses.

What would settle it

Run ActionParty in a game where two identical-looking players are instructed to swap positions in a single step, or where the initial coordinates are deliberately offset for one player; if the video shows the wrong player executing the action, or the state token drifts, the claim that positions suffice for binding is refuted. A second check: test a 3D-style scene with one player temporarily hidden behind an object; the coordinate-only RoPE bias should fail to keep that player's actions bound.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Multi-player game worlds can be simulated as one shared video instead of separate per-player views, avoiding the linear token cost of other approaches.
  • Because subject states are encoded explicitly, identical-looking characters remain distinguishable and can be controlled independently.
  • A single model with a unified action space handles 46 distinct games, so action binding does not require per-game training.
  • The 2D coordinate tokens track players through rollouts, giving a map-level state output alongside the video.
  • Autoregressive generation stays accurate over multiple steps, enabling longer rollouts via a sliding context window.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the coordinate-only state is the load-bearing simplification, the approach will likely struggle when subjects occlude each other or leave the frame; a testable extension is to add a visibility flag or depth coordinate.
  • The same masked-attention recipe could be applied to other binding problems, such as associating multiple text prompts or audio streams with specific objects in generated video.
  • The dependence on externally supplied initial positions suggests a natural follow-up: train a lightweight detector to propose initial state tokens, removing the need for ground-truth initialization at inference.
  • Because actions are abstract buttons whose meaning changes per game, ActionParty effectively learns game dynamics from pixels; this hints that the same architecture could be used for procedural level design or automated playtesting.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ActionParty, an autoregressive video diffusion model for multi-subject action control in 2D game environments. The method augments a Wan2.1-based DiT with subject state tokens representing 2D coordinates, jointly denoised with video latents. Masked self-attention prevents subject-state mixing, masked cross-attention enforces per-subject action binding, and a 3D RoPE bias anchors subject tokens to spatial positions. Training and evaluation on 46 Melting Pot games with up to seven controllable subjects are presented. The authors report large gains over text-only and pretrained baselines on movement accuracy, effect accuracy, and identity preservation, alongside ablations showing each component is necessary.

Significance. If validated, ActionParty is a meaningful step toward multi-agent video world models. The design is conceptually simple and the ablation study gives internal evidence that the subject-state tokens, cross-attention mask, and RoPE bias are each responsible for the reported behavior. The paper also ships a substantial adaptation of the Melting Pot benchmark to a unified action space, which could be reused. However, the quantitative evidence is weakened by metric-validity and statistical-robustness issues, so the strength of the claims currently exceeds what the reported numbers can support.

major comments (4)
  1. [Sec 4.1, Supplementary B] The Movement Accuracy (MA) metric is the primary evidence for the headline claim, but its reliability on generated frames is not established. The paper states that the subject detectors 'achieve near 100% accuracy on raw video frames' (Sec 4.1) and the supplementary describes training linear models on clean ground-truth tiles. Generated frames contain artifacts, blur, and appearance drift; detector accuracy (position and orientation) on generated frames is never reported. Since actions are relative to subject orientation, an orientation error can reclassify 'forward' as another direction and directly change MA. Please report detector accuracy on generated rollouts (separately for position and orientation) or provide human-verified samples.
  2. [Tables 1–3] All quantitative claims rest on point estimates with no error bars, multiple seeds, or significance tests. The abstract says 'significant improvements' but the 0.779 vs 0.158 MA gap in Table 1 could be within run-to-run variance; the evaluation set is 230 rollouts with no repeated generation seeds. Please add confidence intervals (e.g., bootstrap over rollouts or multiple random seeds) and, where appropriate, a paired test between ActionParty and Text-Action on the same input rollouts.
  3. [Sec 4.1, Table 2] Effect Accuracy (EA) relies on an 'empirically set' SSIM threshold of >=0.85 with no sensitivity analysis. This threshold is load-bearing for the claim that ActionParty 'more than doubles' baselines on Interact actions (Table 2: 0.774 vs 0.326). Moreover, EA is computed on a 3x3 patch centered at the ground-truth player position, so any state drift moves the effect out of the patch and is scored as a failure even if the action was correctly bound. The supplementary acknowledges this, but the main text should present a threshold sweep and an analysis of how EA varies with player-position error.
  4. [Tables 1 vs 3] State prediction error (z_t error) is a core quantity: the model's ability to control subjects depends on accurate coordinates for the RoPE bias. Yet z_t error is reported only for the Coins ablation (Table 3), not for the full 46-game benchmark (Table 1). The reader cannot judge whether the state tokens actually track subjects in the general setting. Please report z_t error on the full evaluation set and its correlation with MA. If the initial-position assumption (Sec 3.4) is relaxed, quantify how much position error the method tolerates before action binding degrades.
minor comments (5)
  1. [Sec 4.1] The description of the action space lists 7 base actions but enumerates 8 (idle, 4 moves, 2 turns, and Interact). Clarify whether Interact is outside the 7 base movement actions.
  2. [Sec 4.1] Define the alignment threshold for DR ('percentage of steps where the subject's position aligns with ground-truth' — align to within how many pixels/tiles?).
  3. [Supplementary D] The statement that predicted coordinates 'appear to drift' in long horizons should be moved to a limitation in the main text; it is directly relevant to the 'precise per-subject control' claim.
  4. [General] Minor typos: 'Corr:pondaven@robots.ox.ac.uk' should be 'Correspondence:...'; some 'T able' artifacts in the text.
  5. [Sec 2] The 'first' claim about controlling up to seven players in a single scene should explicitly distinguish from per-subject multi-view methods (Multiverse/Solaris) in the same paragraph to avoid ambiguity.

Circularity Check

0 steps flagged

No significant circularity: the central action-binding claims are evaluated against held-out ground-truth rollouts with externally defined metrics, not against the model's own fitted outputs.

full rationale

ActionParty's derivation is self-contained and empirical. The model objective (Eq. 2) jointly predicts video frames x and subject states z conditioned on actions; z_t is a model output trained with a flow-matching loss against ground-truth Melting Pot coordinates, not a fitted stand-in for the evaluation metrics. The headline metrics are external to z_t: MA uses separately trained subject detectors on generated frames to compare observed movement with the commanded action; EA compares SSIM of a ground-truth-centered 3x3 patch; SP and DR check appearance retention and position alignment with held-out ground truth. None of these metrics is defined as "the model's z_t matches its own video," so success is not tautological. The RoPE bias uses the previous step's state as an architectural input, and the paper's ablations (w/o MSA, w/o MCA, frame-wise MCA, No RoPE) isolate the mechanism's contribution. The paper's self-citations (e.g., Mind the Time, Video Motion Transfer) appear only in related-work motivation and are not the evidence for the main claim. Stated limitations—initial positions must be provided, predicted z may drift, detector accuracy on generated frames is not reported, and the EA SSIM threshold is empirically set—are validity/robustness concerns, not circular reductions.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 1 invented entities

The central method rests on domain assumptions about the Melting Pot environment (2D coordinates as sufficient state, no overlap, available initial positions) and on the pretrained Wan backbone. The free parameters are evaluation thresholds and normalization choices rather than fitted physics. The introduced entity is a latent state token that is internal to the architecture and has no independent external evidence beyond the paper's own benchmark.

free parameters (2)
  • SSIM threshold for Effect Accuracy = 0.85
    The paper 'empirically set a threshold patch SSIM>=0.85' to decide whether an Interact action produced a visible effect. This threshold directly affects every reported EA number and is a hand-set evaluation parameter.
  • Coordinate normalization scale = 4.0
    2D coordinates are re-centered and scaled by 4.0 so they have similar statistics to VAE-encoded video latents (Supp. B). This is a hand-chosen hyperparameter that affects the state-token encoding and training dynamics.
axioms (5)
  • domain assumption Wan2.1-1.3B provides a usable video prior and can absorb additional state tokens via fine-tuning
    The entire method is an extension of this pretrained DiT (Sec. 4.1); if the backbone could not be fine-tuned with concatenated state tokens and masked attention, the claimed capability would not obtain.
  • domain assumption 2D coordinates are a sufficient subject state because two subjects cannot occupy the same position
    Sec. 3.2 defines z_i^t = (h_i^t, w_i^t) and justifies it by the no-overlap property of Melting Pot. This excludes 3D scenes, partial visibility, overlapping subjects, and any state beyond position; orientation and interactions must be inferred from pixels.
  • domain assumption Initial frame and initial subject positions are provided at inference
    Sec. 3.4 explicitly assumes x0 and {z_i^0} are given; without this, identical sprites cannot be distinguished. This is load-bearing for the demonstration and a key limitation for general use.
  • domain assumption LLM-adapted 16x16 maps preserve the dynamics of the original Melting Pot games
    Supp. C: maps are rewritten by an LLM to fit the fixed 512x512 resolution, with playability tested using trained policies. If the adaptation changes game semantics, the training and evaluation distribution is no longer the original Melting Pot benchmark.
  • domain assumption Actions are abstract buttons whose effects depend on environment and subject state
    Sec. 3.1: the same action A_k can mean different things in different games (interact picks up an item vs. opens a door). The model must learn context-dependent dynamics from data rather than from an explicit action semantics.
invented entities (1)
  • Subject state tokens z_i^t no independent evidence
    purpose: Persistent latent identifiers carrying each subject's 2D position; used to bind actions to subjects in cross-attention and to anchor rendering via RoPE in self-attention.
    The paper evaluates predicted z_t against Melting Pot ground truth on its own benchmark, but releases no code, checkpoints, or external validation. The tokens are an internal architectural construct, not an independently measured entity with a falsifiable handle outside this paper.

pith-pipeline@v1.3.0-alltime-deepseek · 16752 in / 12524 out tokens · 111109 ms · 2026-08-03T00:51:39.342266+00:00 · methodology

0 comments
read the original abstract

Recent advances in video diffusion have enabled the development of "world models" capable of simulating interactive environments. However, these models are largely restricted to single-agent settings, failing to control multiple agents simultaneously in a scene. In this work, we tackle a fundamental issue of action binding in existing video diffusion models, which struggle to associate specific actions with their corresponding subjects. For this purpose, we propose ActionParty, an action controllable multi-subject world model for generative video games. It introduces subject state tokens, i.e. latent variables that persistently capture the state of each subject in the scene. By jointly modeling state tokens and video latents with a spatial biasing mechanism, we disentangle global video frame rendering from individual action-controlled subject updates. We evaluate ActionParty on the Melting Pot benchmark, demonstrating the first video world model capable of controlling up to seven players simultaneously across 46 diverse environments. Our results show significant improvements in action-following accuracy and identity consistency, while enabling robust autoregressive tracking of subjects through complex interactions.

Figures

Figures reproduced from arXiv: 2604.02330 by Alexander Pondaven, Aliaksandr Siarohin, Fabio Pizzati, Igor Gilitschenski, Philip Torr, Sergey Tulyakov, Ziyi Wu.

Figure 1
Figure 1. Figure 1: Left: Action binding failure case of text-to-video models with prompt: ‘The red triangle moves right and the blue square moves up. Then the red triangle moves down and the blue square moves left. Then the red triangle moves up and the blue square moves right. Then the red triangle moves left and the blue square moves down.’ Right: ActionParty enables action control of multiple subjects in a scene. demonstr… view at source ↗
Figure 2
Figure 2. Figure 2: ActionParty pipeline. Given initial video frames x0:t and subject states z0:t as context, we aim to generate the next video frame xt+1 conditioned on action inputs a0:t and a text description of the game c. We concatenate the video and subject state tokens along the sequence dimension and feed them into a diffusion transformer (DiT) for joint denoising. Each DiT block first runs self-attention with an atte… view at source ↗
Figure 3
Figure 3. Figure 3: Attention mechanisms in ActionParty DiT. (a) In self-attention, we use RoPE to link a subject in a video frame to its state token z t i . We encode the state token with the subject’s coordinates in the previous timestep, biasing it to attend to video tokens close to the subject. (b) In cross-attention, subject i’s state tokens z i is only allowed to attend to its own actions a i , ensuring correct subject-… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison with baselines. We display ground truth subject posi￾tions and orientations at each step with an arrow for each subject, which reflects the ground truth subjects (notice that the arrow is consistent in all methods). Our method is the only one able to follow the ground truth actions with appropriate action binding. 4.2 Action binding evaluation Quantitative evaluation. We compare acti… view at source ↗
Figure 5
Figure 5. Figure 5: Movement accuracy (MA) over autoregressive steps. ActionParty maintains stable action binding across multiple rollout steps, whereas baselines degrade over time and get close to 0. yield suboptimal yet acceptable results on the first step, it quickly degrades in subsequent steps. Conversely, thanks to the joint modeling of subject states, ActionParty allows for a relatively stable autoregressive inference,… view at source ↗
Figure 6
Figure 6. Figure 6: Results of ActionParty with xt and zt (circle) predictions for 2-, 4-, and 7- player games with a diversity of subjects. The full frames are generated and we show a centred crop around 2 subjects for visualization purposes with action annotations on the resulting frame. Note that actions are relative to orientation, so the subject moving forward is in the direction of where it is facing [PITH_FULL_IMAGE:f… view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparison of ablations run on the same initial frame and action trajectory. The blue subject is still, while the purple one moves and returns to its previ￾ous position. The zt trajectory is plotted for each subject. If we remove our introduced components, we lose the possibility to control characters. 4.3 Ablation studies We conduct ablation studies in a smaller-scale setup due to excessive tr… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Multiplayer Interactive World Models with Representation Autoencoders

    cs.CV 2026-07 accept novelty 7.0

    A 5B-parameter latent diffusion model generates real-time four-player Rocket League matches conditioned on all players' actions, staying stable far beyond its training horizon.

  2. Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends

    cs.CV 2026-05 unverdicted novelty 2.0

    This survey reviews trends, challenges, benchmarks, and future directions in action-conditioned interactive world modeling for video and 3D generation.

Reference graph

Works this paper leans on

94 extracted references · 22 linked inside Pith · cited by 2 Pith papers

  1. [1]

    arXiv preprint arXiv:2211.13746 (2022) 3, 4, 9

    Agapiou, J.P., et al.: Melting Pot 2.0. arXiv preprint arXiv:2211.13746 (2022) 3, 4, 9

    Pith/arXiv arXiv 2022

  2. [2]

    NeurIPS (2024) 4

    Alonso,E.,Jelley,A.,Micheli,V.,Kanervisto,A.,Storkey,A.J.,Pearce,T.,Fleuret, F.: Diffusion for world modeling: Visual details matter in atari. NeurIPS (2024) 4

    2024

  3. [3]

    Ball, P.J., et al.: Genie 3: A new frontier for world models (2025) 1, 2, 4

    2025

  4. [4]

    In: International Symposium on Distributed Autonomous Robotic Systems (2022) 2

    Bettini, M., Kortvelesy, R., Blumenkamp, J., Prorok, A.: VMAS: A vectorized multi-agent simulator for collective robot learning. In: International Symposium on Distributed Autonomous Robotic Systems (2022) 2

    2022

  5. [5]

    In: CVPR (2023) 3

    Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S.W., Fidler, S., Kreis, K.: Align your Latents: High-resolution video synthesis with latent diffusion mod- els. In: CVPR (2023) 3

    2023

  6. [6]

    arXiv preprint arXiv:2311.15127 (2023) 1, 3

    Blattmann, A., et al.: Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 (2023) 1, 3

    Pith/arXiv arXiv 2023

  7. [7]

    OpenAI technical reports (2024),https://openai.com/research/video-generation-models-as- world-simulators1, 3

    Brooks, T., et al.: Video generation models as world simulators. OpenAI technical reports (2024),https://openai.com/research/video-generation-models-as- world-simulators1, 3

    2024

  8. [8]

    In: ICML (2024) 1, 4 16 A

    Bruce, J., et al.: Genie: Generative interactive environments. In: ICML (2024) 1, 4 16 A. Pondaven et al

    2024

  9. [9]

    TOG (2023) 4

    Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L., Cohen-Or, D.: Attend-and-Excite: Attention-based semantic guidance for text-to-image diffusion models. TOG (2023) 4

    2023

  10. [10]

    NeurIPS (2024) 3, 4

    Chen, B., Martí Monsó, D., Du, Y., Simchowitz, M., Tedrake, R., Sitzmann, V.: Diffusion Forcing: Next-token prediction meets full-sequence diffusion. NeurIPS (2024) 3, 4

    2024

  11. [11]

    arXiv preprint arXiv:2310.19512 (2023) 3

    Chen, H., Xia, M., He, Y., Zhang, Y., Cun, X., Yang, S., Xing, J., Liu, Y., Chen, Q., Wang, X., Weng, C., Shan, Y.: VideoCrafter1: Open diffusion models for high- quality video generation. arXiv preprint arXiv:2310.19512 (2023) 3

    Pith/arXiv arXiv 2023

  12. [12]

    In: CVPR (2024) 3

    Chen, H., Zhang, Y., Cun, X., Xia, M., Wang, X., Weng, C., Shan, Y.: VideoCrafter2: Overcoming data limitations for high-quality video diffusion mod- els. In: CVPR (2024) 3

    2024

  13. [13]

    arXiv preprint arXiv:2506.18088 (2025) 2

    Chen, T., Chen, Z., Chen, B., Cai, Z., Liu, Y., Li, Z., Liang, Q., Lin, X., Ge, Y., Gu, Z., et al.: RoboTwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088 (2025) 2

    Pith/arXiv arXiv 2025

  14. [14]

    In: ECCV (2024) 4

    Chi, S., Chi, H.g., Ma, H., Agarwal, N., Siddiqui, F., Ramani, K., Lee, K.: M2D2M: Multi-motion generation from text with discrete diffusion models. In: ECCV (2024) 4

    2024

  15. [15]

    Decart, E., McIntyre, Q., Campbell, S., Chen, X., Wachen, R.: Oasis: A universe in a transformer (2024),https://oasis-model.github.io/4

    2024

  16. [16]

    arXiv preprint arXiv:2412.05496 (2024) 9

    Dong,J.,Feng,B.,Guessous,D.,Liang,Y.,He,H.:FlexAttention:Aprogramming model for generating optimized attention kernels. arXiv preprint arXiv:2412.05496 (2024) 9

    Pith/arXiv arXiv 2024

  17. [17]

    Enigma team: Introducing multiverse: The first ai multiplayer world model (2025), https://enigma.inc/blog4, 9

    2025

  18. [18]

    In: ICML (2024) 6, 9

    Esser, P., et al.: Scaling rectified flow transformers for high-resolution image syn- thesis. In: ICML (2024) 6, 9

    2024

  19. [19]

    In: ICCV Workshops (2023) 4

    Farshad, A., Yeganeh, Y., Chi, Y., Shen, C., Ommer, B., Navab, N.: Scenegenie: Scene graph guided diffusion models for image synthesis. In: ICCV Workshops (2023) 4

    2023

  20. [20]

    In: CVPR (2024) 4

    Gao, G., Liu, W., Chen, A., Geiger, A., Schölkopf, B.: Graphdreamer: Composi- tional 3d scene synthesis from scene graphs. In: CVPR (2024) 4

    2024

  21. [21]

    In: CVPR (2025) 4

    Geng, D., et al.: Motion Prompting: Controlling video generation with motion trajectories. In: CVPR (2025) 4

    2025

  22. [22]

    NeurIPS (2025) 4

    Gokmen, A.B., Ekin, Y., Bilecen, B.B., Dundar, A.: RoPECraft: Training-free motiontransferwithtrajectory-guidedropeoptimizationondiffusiontransformers. NeurIPS (2025) 4

    2025

  23. [23]

    arXiv preprint arXiv:2012.05208 (2020) 4

    Greff, K., Van Steenkiste, S., Schmidhuber, J.: On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208 (2020) 4

    Pith/arXiv arXiv 2012

  24. [24]

    In: ICLR (2024) 3

    Guo, Y., Yang, C., Rao, A., Wang, Y., Qiao, Y., Lin, D., Dai, B.: Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In: ICLR (2024) 3

    2024

  25. [25]

    arXiv preprint arXiv:2312.06662 (2023) 3

    Gupta, A., Yu, L., Sohn, K., Gu, X., Hahn, M., Fei-Fei, L., Essa, I., Jiang, L., Lezama, J.: Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662 (2023) 3

    Pith/arXiv arXiv 2023

  26. [26]

    NeurIPS (2018) 3

    Ha, D., Schmidhuber, J.: Recurrent world models facilitate policy evolution. NeurIPS (2018) 3

    2018

  27. [27]

    arXiv preprint arXiv:1803.10122 (2018) 1, 3 ActionParty 17

    Ha, D., Schmidhuber, J.: World Models. arXiv preprint arXiv:1803.10122 (2018) 1, 3 ActionParty 17

    Pith/arXiv arXiv 2018

  28. [28]

    In: ICLR (2020) 4

    Hafner, D., Lillicrap, T., Ba, J., Norouzi, M.: Dream to control: Learning behaviors by latent imagination. In: ICLR (2020) 4

    2020

  29. [29]

    arXiv preprint arXiv:2508.13009 (2025) 3, 4

    He,X.,etal.:Matrix-Game2.0:Anopen-sourcereal-timeandstreaminginteractive world model. arXiv preprint arXiv:2508.13009 (2025) 3, 4

    Pith/arXiv arXiv 2025

  30. [30]

    In: NeurIPS (2020) 1, 3

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: NeurIPS (2020) 1, 3

    2020

  31. [31]

    NeurIPS (2022) 1, 3

    Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., Fleet, D.J.: Video diffusion models. NeurIPS (2022) 1, 3

    2022

  32. [32]

    arXiv preprint arXiv:2210.02303 (2022) 1, 3

    Ho, J., et al.: Imagen Video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 (2022) 1, 3

    Pith/arXiv arXiv 2022

  33. [33]

    arXiv preprint arXiv:2512.04040 (2025) 1, 3

    Hong, Y., et al.: RELIC: Interactive video world model with long-horizon memory. arXiv preprint arXiv:2512.04040 (2025) 1, 3

    arXiv 2025

  34. [34]

    arXiv preprint arXiv:2309.17080 (2023) 2

    Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., Corrado, G.: GAIA-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080 (2023) 2

    Pith/arXiv arXiv 2023

  35. [35]

    PAMI (2025) 4

    Huang, K., Duan, C., Sun, K., Xie, E., Li, Z., Liu, X.: T2I-CompBench++: An en- hanced and comprehensive benchmark for compositional text-to-image generation. PAMI (2025) 4

    2025

  36. [36]

    NeurIPS (2025) 3, 4

    Huang, X., Li, Z., He, G., Zhou, M., Shechtman, E.: Self Forcing: Bridging the train-test gap in autoregressive video diffusion. NeurIPS (2025) 3, 4

    2025

  37. [37]

    In: CVPR (2024) 4

    Jeong, H., Park, G.Y., Ye, J.C.: VMC: Video motion customization using temporal attention adaption for text-to-video diffusion models. In: CVPR (2024) 4

    2024

  38. [38]

    NeurIPS (2023) 4

    Jiang, B., Chen, X., Liu, W., Yu, J., Yu, G., Chen, T.: MotionGPT: Human motion as a foreign language. NeurIPS (2023) 4

    2023

  39. [39]

    In: ECCV (2024) 4

    Jin, P., Li, H., Cheng, Z., Li, K., Yu, R., Liu, C., Ji, X., Yuan, L., Chen, j.: Local action-guided motion diffusion model for text-to-motion generation. In: ECCV (2024) 4

    2024

  40. [40]

    arXiv preprint arXiv:1412.6980 (2014) 8

    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014) 8

    Pith/arXiv arXiv 2014

  41. [41]

    arXiv preprint arXiv:2412.03603 (2024) 3

    Kong, W., et al.: HunyuanVideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603 (2024) 3

    Pith/arXiv arXiv 2024

  42. [42]

    In: CVPR (2023) 2

    Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., Lee, Y.J.: GLIGEN: Open-set grounded text-to-image generation. In: CVPR (2023) 2

    2023

  43. [43]

    In: ICLR (2024) 2, 4

    Lian, L., Shi, B., Yala, A., Darrell, T., Li, B.: Llm-grounded video diffusion models. In: ICLR (2024) 2, 4

    2024

  44. [44]

    In: COLM (2024) 4

    Lin, H., Zala, A., Cho, J., Bansal, M.: VideoDirectorGPT: Consistent multi-scene video generation via llm-guided planning. In: COLM (2024) 4

    2024

  45. [45]

    In: ICLR (2023) 8

    Lipman, Y., Chen, R.T., Ben-Hamu, H., Nickel, M., Le, M.: Flow matching for generative modeling. In: ICLR (2023) 8

    2023

  46. [46]

    In: ECCV (2022) 4

    Liu, N., Li, S., Du, Y., Torralba, A., Tenenbaum, J.B.: Compositional visual gen- eration with composable diffusion models. In: ECCV (2022) 4

    2022

  47. [47]

    In: ICLR (2023) 8

    Liu, X., et al.: Flow straight and fast: Learning to generate and transfer data with rectified flow. In: ICLR (2023) 8

    2023

  48. [48]

    In: ICCV (2025) 4

    Liu, Y., Li, X., Zhang, Y., Qi, L., Li, X., Wang, W., Li, C., Li, X., Yang, M.H.: Controllable 3d outdoor scene generation via scene graphs. In: ICCV (2025) 4

    2025

  49. [49]

    arXiv preprint arXiv:2502.10248 (2025) 3

    Ma, G., et al.: Step-Video-T2V technical report: The practice, challenges, and future of video foundation model. arXiv preprint arXiv:2502.10248 (2025) 3

    Pith/arXiv arXiv 2025

  50. [50]

    In: CVPR (2022) 3 18 A

    Menapace, W., Lathuiliere, S., Siarohin, A., Theobalt, C., Tulyakov, S., Golyanik, V., Ricci, E.: Playable Environments: Video manipulation in space and time. In: CVPR (2022) 3 18 A. Pondaven et al

    2022

  51. [51]

    In: CVPR (2021) 3

    Menapace, W., Lathuiliere, S., Tulyakov, S., Siarohin, A., Ricci, E.: Playable video generation. In: CVPR (2021) 3

    2021

  52. [52]

    CVPR (2024) 3

    Menapace, W., et al.: Snap video: Scaled spatiotemporal transformers for text-to- video synthesis. CVPR (2024) 3

    2024

  53. [53]

    In: ICLR (2025) 4

    Namekata, K., Bahmani, S., Wu, Z., Kant, Y., Gilitschenski, I., Lindell, D.B.: SG-I2V. In: ICLR (2025) 4

    2025

  54. [54]

    In: ICCV (2025) 4

    Pardo, A., Pizzati, F., Zhang, T., Pondaven, A., Torr, P., Perez, J.C., Ghanem, B.: Matchdiffusion: Training-free generation of match-cuts. In: ICCV (2025) 4

    2025

  55. [55]

    ICCV (2023) 3

    Peebles, W., Xie, S.: Scalable diffusion models with transformers. ICCV (2023) 3

    2023

  56. [56]

    arXiv preprint arXiv:2603.06679 (2026) 4

    Po, R., Zhang, D.J., Hertz, A., Wetzstein, G., Wadhwa, N., Ruiz, N.: Multi- gen: Level-design for editable multiplayer worlds in diffusion game engines. arXiv preprint arXiv:2603.06679 (2026) 4

    arXiv 2026

  57. [57]

    arXiv preprint arXiv:2410.13720 (2024) 3

    Polyak, A., et al.: Movie Gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720 (2024) 3

    Pith/arXiv arXiv 2024

  58. [58]

    In: CVPR (2025) 4

    Pondaven, A., Siarohin, A., Tulyakov, S., Torr, P., Pizzati, F.: Video motion trans- fer with diffusion transformers. In: CVPR (2025) 4

    2025

  59. [59]

    NeurIPS (2023) 4

    Rassin, R., Hirsch, E., Glickman, D., Ravfogel, S., Goldberg, Y., Chechik, G.: Lin- guistic Binding in Diffusion Models: Enhancing attribute correspondence through attention map alignment. NeurIPS (2023) 4

    2023

  60. [60]

    In: MICCAI (2015) 3

    Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomed- ical image segmentation. In: MICCAI (2015) 3

    2015

  61. [61]

    arXiv preprint arXiv:2602.22208 (2026) 4, 9

    Savva, G., Michel, O., Lu, D., Waiwitlikhit, S., Meehan, T., Mishra, D., Poddar, S., Lu, J., Xie, S.: Solaris: Building a multiplayer video world model in minecraft. arXiv preprint arXiv:2602.22208 (2026) 4, 9

    arXiv 2026

  62. [62]

    Seawead, T., Yang, C., et al.: Seaweed-7B: Cost-effective training of video genera- tion foundation model (2025) 3

    2025

  63. [63]

    Sharma, A., et al.: Veo (2024),https://deepmind.google/technologies/veo/2

    2024

  64. [64]

    In: ICML (2015) 3

    Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsuper- vised learning using nonequilibrium thermodynamics. In: ICML (2015) 3

    2015

  65. [65]

    Neurocomputing (2024) 3

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., Liu, Y.: Roformer: Enhanced trans- former with rotary position embedding. Neurocomputing (2024) 3

    2024

  66. [66]

    In: CVPR (2021) 2

    Suo, S., Regalado, S., Casas, S., Urtasun, R.: TrafficSim: Learning to simulate realistic multi-agent behaviors. In: CVPR (2021) 2

    2021

  67. [67]

    In: ICLR (2023) 4

    Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-Or, D., Bermano, A.H.: Human motion diffusion model. In: ICLR (2023) 4

    2023

  68. [68]

    arXiv preprint arXiv:1812.01717 (2018) 10

    Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., Gelly, S.: Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717 (2018) 10

    Pith/arXiv arXiv 2018

  69. [69]

    In: ICLR (2025) 1

    Valevski, D., Leviathan, Y., Arar, M., Fruchter, S.: Diffusion models are real-time game engines. In: ICLR (2025) 1

    2025

  70. [70]

    In: ICLR (2025) 4

    Valevski, D., Leviathan, Y., Arar, M., Fruchter, S.: Diffusion models are real-time game engines. In: ICLR (2025) 4

    2025

  71. [71]

    arXiv preprint arXiv:2503.20314 (2025) 1, 3, 6, 8, 9

    Wang, A., et al.: Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314 (2025) 1, 3, 6, 8, 9

    Pith/arXiv arXiv 2025

  72. [72]

    arXiv preprint arXiv:2402.01566 (2024) 4

    Wang, J., Zhang, Y., Zou, J., Zeng, Y., Wei, G., Yuan, L., Li, H.: Boxima- tor: Generating rich and controllable motions for video synthesis. arXiv preprint arXiv:2402.01566 (2024) 4

    Pith/arXiv arXiv 2024

  73. [73]

    arXiv preprint arXiv:2309.04615 (2023) 4

    Wang, Z., Meger, D.: VDFD: Multi-agent value decomposition framework with disentangled world model. arXiv preprint arXiv:2309.04615 (2023) 4

    arXiv 2023

  74. [74]

    arXiv preprint arXiv:2602.16229 (2026) 4 ActionParty 19

    Wang, Z., Shi, C., Hu, J., Rohling, K., Martín-Martín, R., Zhang, A., Stone, P.: Factored latent action world models. arXiv preprint arXiv:2602.16229 (2026) 4 ActionParty 19

    Pith/arXiv arXiv 2026

  75. [75]

    In: ECCV (2024) 4

    Wu, W., Li, Z., Gu, Y., Zhao, R., He, Y., Zhang, D.J., Shou, M.Z., Li, Y., Gao, T., Zhang, D.: DragAnything: Motion control for anything using entity representation. In: ECCV (2024) 4

    2024

  76. [76]

    NeurIPS (2024) 4

    Wu, Z., Rubanova, Y., Kabra, R., Hudson, D.A., Gilitschenski, I., Aytar, Y., Van Steenkiste, S., Allen, K.R., Kipf, T.: Neural Assets: 3d-aware multi-object scene synthesis with image diffusion models. NeurIPS (2024) 4

    2024

  77. [77]

    In: CVPR (2025) 2, 4

    Wu, Z., Siarohin, A., Menapace, W., Skorokhodov, I., Fang, Y., Chordia, V., Gilitschenski, I., Tulyakov, S.: Mind the Time: Temporally-controlled multi-event video generation. In: CVPR (2025) 2, 4

    2025

  78. [78]

    NeurIPS (2025) 4

    Xiao, Z., Lan, Y., Zhou, Y., Ouyang, W., Yang, S., Zeng, Y., Pan, X.: WorldMem: Long-term consistent world simulation with memory. NeurIPS (2025) 4

    2025

  79. [79]

    NeurIPS (2024) 4

    Xiao, Z., Zhou, Y., Yang, S., Pan, X.: Video diffusion models are training-free motion interpreter and controller. NeurIPS (2024) 4

    2024

  80. [80]

    In: ECCV (2024) 3

    Xing, J., Xia, M., Zhang, Y., Chen, H., Wang, X., Wong, T.T., Shan, Y.: Dynam- iCrafter: Animating open-domain images with video diffusion priors. In: ECCV (2024) 3

    2024

Showing first 80 references.