Pith. sign in

REVIEW 3 major objections 5 minor 94 references

Hierarchical schema-guided transitions let multimodal models jointly predict and simulate visual dynamics, yielding more consistent and controllable narratives.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 21:35 UTC pith:Q2BZBYQT

load-bearing objection Solid empirical recipe for hierarchical visual dynamics that actually moves the needle on consistency and controllability; worth engaging. the 3 major comments →

arxiv 2607.04112 v1 pith:Q2BZBYQT submitted 2026-07-05 cs.LG cs.AIcs.CLcs.CV

DynaVieW: Schema-Guided World Modeling for Understanding Hierarchical Visual Dynamics

Silin Gao , Hao Zhao , Zeming Chen , Sepideh Mamooler , Antara Raaghavi Bhattacharya , Qiyu Wu , Hiromi Wakaki , Yuki Mitsufuji
show 3 more authors
Li Mi Syrielle Montariol Antoine Bosselut
This is my paper
classification cs.LG cs.AIcs.CLcs.CV
keywords world modelsvisual dynamicsinterleaved vision-language modelshierarchical schemamixture-of-expertsvisual narrative generationinstruction following
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multimodal large language models still fail to track how visual scenes evolve over time: they cannot reliably predict the multi-level constituents of change (actions, object motions, spatial relations, camera moves) or use those predictions to generate coherent next frames. DynaVieW treats the problem as continued pre-training on interleaved state-transition sequences. States are sharp, diverse video keyframes; transitions are hierarchical JSON descriptions that separately name high-level activities, sub-activities, atomic actions, and seven classes of low-level transformations together with their contributions. The model learns both next-transition prediction and next-state image simulation under a mixture-of-experts architecture, selective attention that drops redundant history, and a re-weighted loss that prevents the rigid schema tokens from dominating the flexible slot values. On visual narrative generation and instruction-following world-simulation benchmarks the resulting model produces sequences that are more consistent across scenes, more steerable by fine-grained transition prompts, and more faithful to action instructions than strong interleaved baselines.

Core claim

Jointly training a mixture-of-experts multimodal model on interleaved visual states and hierarchical schema-structured transitions, using selective attention and schema-token re-weighting, produces a world model whose downstream visual narratives and action-conditioned simulations are measurably more consistent, controllable, and instruction-following than models trained without that hierarchical dynamic supervision.

What carries the argument

Interleaved state-transition sequences: video keyframes serve as states; hierarchical JSON schemas serve as transitions that separately encode high-level activity, sub-activities, atomic actions, seven transformation types, and their contributions; these sequences are learned jointly under selective multimodal attention and a schema-token re-weighted cross-entropy loss.

Load-bearing premise

The hierarchical JSON schemas produced by a large vision-language model under constrained decoding, together with the fixed keyframe sharpness and similarity thresholds, are assumed to be a complete and unbiased enough description of real-world visual dynamics for generalizable world modeling rather than schema-specific artifacts.

What would settle it

Train an otherwise identical model on the same keyframes but with natural-language or randomly permuted transition descriptions; if the measured gains on VinaBench consistency and LEGO instruction-following disappear, the hierarchical schema itself is not the causal driver.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Visual narrative systems can accept external hierarchical transition prompts and still produce coherent, style-consistent image sequences.
  • World-simulation agents can follow fine-grained action instructions with higher visual fidelity because the model has already practiced predicting the same multi-level changes.
  • Error accumulation across long image sequences is reduced once intermediate transitions are forced to name both high-level progress and low-level spatial/motion facts.
  • The same selective-attention and re-weighted-loss recipe can be reused for any interleaved vision-language model that needs to learn long dynamic sequences without copying earlier text.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the hierarchical schema is the main source of the gains, then lighter or automatically induced schemas may achieve similar controllability at lower annotation cost.
  • The same state-transition formulation could be applied to longer video generation by treating every few seconds as a new state and predicting the intervening transition.
  • Selective attention that systematically forgets earlier transitions may also help non-visual sequence models that must avoid rote copying of earlier structured text.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces DynaVieW, a mixture-of-Transformer-experts world model continued-pretrained on interleaved visual state–transition sequences. States are keyframes extracted from Ego4D, AgiBotWorld-Alpha and ShareGPT4Video; transitions are hierarchical JSON schemas (high-level activity → sub-activities → atomic actions → seven transformation types + three contribution types) annotated by InternVL-78B under constrained decoding. Joint training uses selective multimodal attention (dropout on earlier states, masking of earlier transitions) and a schema-token re-weighted CE loss (weight 0.1 on structural tokens) plus diffusion MSE for state simulation. Downstream, zero-shot and SFT results on VinaBench (visual narrative) and LEGO (instruction-following world simulation) show gains in consistency, controllability and instruction following over BAGEL, Story2Board and other interleaved baselines, supported by GPT-4o/Gemini judges, human preference (47.6 % vs 35.4 %), ablations (Figure 4) and internal validation (Table 1).

Significance. If the empirical gains hold under broader scrutiny, the work supplies a concrete, reproducible recipe—hierarchical schema-guided transitions + selective attention + re-weighted loss—for elevating interleaved multimodal LMs on visual dynamics. Strengths include multi-source video coverage, explicit ablations isolating each component, dual VLM judges plus human preference, external benchmarks (VinaBench, LEGO), and public code/data release. The approach is of clear interest to the multimodal and world-modeling communities; the main open question is how far the schema and short training windows generalize beyond the reported settings.

major comments (3)
  1. [Section 2.1 / Appendix A.1] Section 2.1 and Appendix A.1: The hierarchical JSON schema is author-defined and populated by InternVL-78B under constrained decoding. Table 1 shows low Reject rates for both gold and predicted transitions, and Figure 4 ablations confirm that removing JSON structure or low-level slots hurts VinaBench. Nevertheless, the paper does not quantify annotation bias (e.g., systematic under- or over-reporting of particular transformation categories) or test whether an alternative schema yields comparable gains. A short sensitivity analysis or inter-annotator comparison with a second VLM would strengthen the claim that the schema is a sufficiently complete and unbiased representation of visual dynamics rather than a model-specific artifact.
  2. [Appendix A.2 / Section 4] Appendix A.2 and Section 4: Training sequences are split to a maximum of 6 states (sliding window stride 3); VinaBench storyboards are likewise short (typically 5–9 frames). The paper claims improved long-horizon consistency, yet the reported experiments do not stress-test sequences substantially longer than the training window. Explicit discussion of this scope limitation, and ideally a short scaling experiment or failure-mode analysis on longer roll-outs, would make the consistency claims more precise.
  3. [Table 3 / Section 4] Table 3 (controllability): Transitions are supplied by Gemma-3 rather than DynaVieW itself, which cleanly isolates steerability. The paper notes that Gemma-3 is not itself a world model and that noisy transitions are intentional. Still, the large gap versus BAGEL could partly reflect format familiarity (DynaVieW was pre-trained on the same JSON schema). Reporting a control in which BAGEL is also fine-tuned or prompted with the identical schema format (beyond the NL conversion already shown) would more cleanly attribute the gain to hierarchical modeling rather than schema familiarity.
minor comments (5)
  1. [Figures 1, 3] Figure 1 and Figure 3 captions are dense; a short legend distinguishing understanding vs generation expert pathways would improve readability.
  2. [Tables 2, 6] Table 2 and Table 6: “Non-Char. Ent.”, “Char. Num.” abbreviations are expanded only in the caption; defining them once in the main text would help.
  3. [Section 2.1] Appendix A.1 prompts are long and useful; a one-sentence pointer in the main text to the exact constrained-decoding schema would aid reproducibility without forcing readers into the appendix.
  4. [Introduction / throughout] Minor typographical inconsistencies appear (e.g., “DynaView” vs “DynaVieW” in one place in the introduction; “V AE” spacing). A final proof-read pass would clean these.
  5. [Section 5] Related Work (Section 5) could more explicitly contrast the hierarchical schema against the coarser captions used by BAGEL and related video-interleaved models, to sharpen the novelty claim.

Circularity Check

0 steps flagged

No significant circularity: empirical world-model paper whose claims rest on external benchmarks and ablations, not on self-definitional or fitted-as-prediction reductions.

full rationale

DynaVieW is a continued-pretraining + SFT system paper. The load-bearing claims (better consistency/controllability/instruction-following on VinaBench and LEGO) are measured against held-out external benchmarks, multiple VLM judges, human preference, and ablations that remove selective attention, schema re-weighting, JSON structure, or low-level slots (Figure 4, Tables 2–4, 6–7). State-transition data are constructed from third-party videos (Ego4D, AgiBotWorld-Alpha, ShareGPT4Video) with an external oracle VLM (InternVL-78B) under constrained decoding; the model is not scored solely on reproducing its own annotations. Architecture (MoT, selective attention, schema-token re-weighted CE) is adopted/adapted from BAGEL and standard practice, not justified by a self-citation uniqueness theorem that forbids alternatives. No equation or “prediction” reduces by construction to a fitted parameter or to a definition that already contains the target. Self-citation of the authors’ VinaBench benchmark is ordinary evaluation practice and does not force the reported gains. Therefore the derivation chain is self-contained against external evidence; circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 2 invented entities

The central empirical claim rests on a handful of hand-chosen thresholds and architectural constants, on the assumption that InternVL-generated hierarchical schemas faithfully capture visual dynamics, and on the invented hierarchical schema itself as the organizing representation. No free parameters are fitted to the final evaluation metrics; the listed numbers are design choices fixed before evaluation.

free parameters (4)
  • CLIP cosine similarity threshold for keyframe selection = 0.925
    Fixed at 0.925; controls which candidate frames become states (Section 2.1).
  • Attention dropout rate on earlier states = 0.3
    Fixed at 0.3; part of the selective attention mask (Section 2.2).
  • Schema-token CE loss weight = 0.1
    Fixed at 0.1 versus 1.0 for slot-filling tokens (Section 2.2).
  • Maximum sub-sequence length / sliding-window stride = 6 / 3
    6 states / stride 3 chosen to fit GPU memory (Appendix A.2).
axioms (3)
  • ad hoc to paper A hierarchical JSON schema enumerating high-level activity, sub-activities, atomic actions, seven transformation categories, and three contribution types is a sufficient and non-redundant description of visual dynamics for world-model pretraining.
    Introduced in Section 2.1 and used throughout data construction and training; no external proof of completeness is given.
  • domain assumption InternVL-78B-Instruct under constrained decoding produces accurate, non-hallucinated transition annotations that can serve as gold supervision.
    Stated in Section 2.1 and validated only by GPT-4o agreement rates in Table 1.
  • domain assumption Mixture-of-Transformer-experts with shared selective attention is an appropriate architecture for joint transition prediction and state simulation.
    Inherited from BAGEL and retained without alternative architecture search (Section 2.2).
invented entities (2)
  • Hierarchical visual-dynamics JSON schema (activities → atomic actions → transformations + contributions) no independent evidence
    purpose: Provides structured intermediate targets that force the model to verbalize multi-level dynamics before simulating the next image.
    Defined de novo in Section 2.1; no prior literature uses this exact multi-level schema for interleaved world-model training.
  • DynaVieW model (MoT + selective attention + schema re-weighting) no independent evidence
    purpose: Concrete realization of the schema-guided world-modeling objective.
    The trained system whose downstream gains constitute the paper's main result.

pith-pipeline@v1.1.0-grok45 · 38010 in / 2854 out tokens · 25633 ms · 2026-07-11T21:35:47.130331+00:00 · methodology

0 comments
read the original abstract

Multimodal LLMs struggle to systematically model the temporal evolution of visual scenes in videos or multi-image sequences. Such inputs require models to predict or simulate multiple levels of dynamic constituents, such as actions taken in the visual sequence, and the associated changes to the visual environment that result. To address this challenge, we propose a dynamic schema-guided world model, DynaVieW, optimized for visual dynamic prediction and simulation. DynaVieW achieves an in-depth understanding of visual dynamics by learning interleaved state-transition sequences, where states cover broad visual scenes from video keyframes, and transitions capture comprehensive dynamic constituents within a hierarchical schema. DynaVieW jointly models transition prediction and state simulation under a mixture-of-experts architecture, with a cross-expert selective attention and a schema token re-weighted loss, to ensure effective and robust learning. DynaVieW's understanding of visual dynamics boosts its downstream performance in visual narrative creation and world simulation, showing improved consistency, controllability, and instruction-following.

Figures

Figures reproduced from arXiv: 2607.04112 by Antara Raaghavi Bhattacharya, Antoine Bosselut, Hao Zhao, Hiromi Wakaki, Li Mi, Qiyu Wu, Sepideh Mamooler, Silin Gao, Syrielle Montariol, Yuki Mitsufuji, Zeming Chen.

Figure 1
Figure 1. Figure 1: Overview of DynaVieW. We formulate our world mod￾eling task as learning interleaved state-transition sequences. In particular, DynaVieW jointly learns the prediction of hierarchical transition descriptions (blue) and the simulation of visual states (green) in an alternating manner. No longer purely limited to natural language processing, they perform multimodal understanding and generation, en￾abling abili… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of DynaVieW data construction. We extract video keyframes as our visual states, ensuring high image sharp￾ness and low CLIP embedding similarity across selected keyframes. Based on that, we prompt an oracle VLM (InternVL-78B-Instruct) to analyze the transitions between adjacent states. tent is significantly different from its preceding selected keyframe.3 Specifically, as shown in [PITH_FULL_IMAG… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of DynaVieW modeling approach. The architecture of DynaVieW adopts a mixture of world understanding and generation experts, with shared multimodal selective attention. DynaVieW is jointly trained on the transition prediction and the state simulation, with a schema token re-weighted cross-entropy (CE) loss and a diffusion mean squared error (MSE) loss, respectively. schema. To ensure that the outpu… view at source ↗
Figure 4
Figure 4. Figure 4: Ablation study of each major component in DynaVieW on VinaBench. We (a) remove the multimodal selective attention (w/o Selective Attention), (b) remove the schema token re-weighting of CE loss (w/o Re-weighted Loss), (c) use a heuristic template to translate the JSON-schema transitions into natural language descriptions (w/o JSON-Schema), and (d) use a more coarse-grained transition schema without describi… view at source ↗
Figure 5
Figure 5. Figure 5: An example of DynaVieW state transition. We frame our transition based on a hierarchical JSON schema. A.3. DynaVieW Validation Details We use GPT-4o (Hurst et al., 2024; Achiam et al., 2023) as a VLM judge to validate DynaVieW’s performances of transition prediction and state simulation, and meanwhile to verify the quality of gold transitions and states constructed by our pipeline, based on the 900 validat… view at source ↗
Figure 6
Figure 6. Figure 6: Example storyboards generated by DynaVieW and BAGEL after SFT, along with ground truth storyboards from VinaBench. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: World simulation outputs generated by DynaVieW and BAGEL after SFT, along with ground truth visual answers from LEGO. C.4. Compute Cost and Latency of DynaVieW We have tested the compute and latency costs of models on the downstream visual narrative generation task using a single NVIDIA H100 GPU, averaging results over 300 random testing samples from VinaBench dataset. The latency (in seconds) and the appr… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

94 extracted references · 31 linked inside Pith

  1. [1]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Ego4d: Around the world in 3,000 hours of egocentric video , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  2. [2]

    arXiv preprint arXiv:2503.06669 , year=

    Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems , author=. arXiv preprint arXiv:2503.06669 , year=

    Pith/arXiv arXiv

  3. [3]

    Advances in Neural Information Processing Systems , volume=

    Sharegpt4video: Improving video understanding and generation with better captions , author=. Advances in Neural Information Processing Systems , volume=

  4. [4]

    arXiv preprint arXiv:2505.14683 , year=

    Emerging properties in unified multimodal pretraining , author=. arXiv preprint arXiv:2505.14683 , year=

    Pith/arXiv arXiv

  5. [5]

    Electronic Imaging , volume=

    Perceptual image quality assessment using a normalized Laplacian pyramid , author=. Electronic Imaging , volume=. 2016 , publisher=

    2016

  6. [6]

    International conference on machine learning , pages=

    Learning transferable visual models from natural language supervision , author=. International conference on machine learning , pages=. 2021 , organization=

    2021

  7. [7]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  8. [8]

    Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search , author=. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  9. [9]

    arXiv preprint arXiv:2502.14786 , year=

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features , author=. arXiv preprint arXiv:2502.14786 , year=

    Pith/arXiv arXiv

  10. [10]

    Advances in Neural Information Processing Systems , volume=

    Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution , author=. Advances in Neural Information Processing Systems , volume=

  11. [11]

    arXiv preprint arXiv:2412.15115 , year=

    Qwen2.5 Technical Report , author=. arXiv preprint arXiv:2412.15115 , year=

    Pith/arXiv arXiv

  12. [12]

    2024 , howpublished=

    Black Forest Labs , title=. 2024 , howpublished=

    2024

  13. [13]

    1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space , author=

    FLUX. 1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space , author=. arXiv preprint arXiv:2506.15742 , year=

    Pith/arXiv arXiv

  14. [14]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Vinabench: Benchmark for faithful and consistent visual narratives , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  15. [15]

    European Conference on Computer Vision , pages=

    LEGO: L earning EGO centric Action Frame Generation via Visual Instruction Tuning , author=. European Conference on Computer Vision , pages=

  16. [16]

    Proceedings of the 36th International Conference on Neural Information Processing Systems , pages=

    Egocentric video-language pretraining , author=. Proceedings of the 36th International Conference on Neural Information Processing Systems , pages=

  17. [17]

    Advances in neural information processing systems , volume=

    Gans trained by a two time-scale update rule converge to a local nash equilibrium , author=. Advances in neural information processing systems , volume=

  18. [18]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    The unreasonable effectiveness of deep features as a perceptual metric , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

  19. [19]

    arXiv preprint arXiv:1602.07360 , year=

    SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 0.5 MB model size , author=. arXiv preprint arXiv:1602.07360 , year=

    Pith/arXiv arXiv

  20. [20]

    arXiv preprint arXiv:2202.00512 , year=

    Progressive distillation for fast sampling of diffusion models , author=. arXiv preprint arXiv:2202.00512 , year=

    Pith/arXiv arXiv

  21. [21]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  22. [22]

    arXiv preprint arXiv:2303.08774 , year=

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    Pith/arXiv arXiv

  23. [23]

    arXiv preprint arXiv:2410.21276 , year=

    Gpt-4o system card , author=. arXiv preprint arXiv:2410.21276 , year=

    Pith/arXiv arXiv

  24. [24]

    arXiv preprint arXiv:2601.03267 , year=

    OpenAI GPT-5 System Card , author=. arXiv preprint arXiv:2601.03267 , year=

    Pith/arXiv arXiv

  25. [25]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  26. [26]

    arXiv preprint arXiv:2507.06261 , year=

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year=

    Pith/arXiv arXiv

  27. [27]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Mvbench: A comprehensive multi-modal video understanding benchmark , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  28. [28]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Openeqa: Embodied question answering in the era of foundation models , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  29. [29]

    arXiv preprint arXiv:2503.22152 , year=

    Egotom: Benchmarking theory of mind reasoning from egocentric videos , author=. arXiv preprint arXiv:2503.22152 , year=

    Pith/arXiv arXiv

  30. [30]

    arXiv preprint arXiv:2410.18072 , year=

    Worldsimbench: Towards video generation models as world simulators , author=. arXiv preprint arXiv:2410.18072 , year=

    Pith/arXiv arXiv

  31. [31]

    arXiv preprint arXiv:2506.21876 , year=

    Do vision-language models have internal world models? towards an atomic evaluation , author=. arXiv preprint arXiv:2506.21876 , year=

    Pith/arXiv arXiv

  32. [32]

    arXiv preprint arXiv:2504.00983 , year=

    Worldscore: A unified evaluation benchmark for world generation , author=. arXiv preprint arXiv:2504.00983 , year=

    arXiv

  33. [33]

    arXiv preprint arXiv:2505.24862 , year=

    Vistorybench: Comprehensive benchmark suite for story visualization , author=. arXiv preprint arXiv:2505.24862 , year=

    arXiv

  34. [34]

    arXiv preprint arXiv:2505.02567 , year=

    Unified multimodal understanding and generation models: Advances, challenges, and opportunities , author=. arXiv preprint arXiv:2505.02567 , year=

    arXiv

  35. [35]

    Advances in Neural Information Processing Systems , volume=

    Conceptmix: A compositional image generation benchmark with controllable difficulty , author=. Advances in Neural Information Processing Systems , volume=

  36. [36]

    Advances in neural information processing systems , volume=

    Flamingo: a visual language model for few-shot learning , author=. Advances in neural information processing systems , volume=

  37. [37]

    Advances in Neural Information Processing Systems , volume=

    Multimodal c4: An open, billion-scale corpus of images interleaved with text , author=. Advances in Neural Information Processing Systems , volume=

  38. [38]

    Advances in Neural Information Processing Systems , volume=

    Obelics: An open web-scale filtered dataset of interleaved image-text documents , author=. Advances in Neural Information Processing Systems , volume=

  39. [39]

    The Twelfth International Conference on Learning Representations , year=

    Emu: Generative Pretraining in Multimodality , author=. The Twelfth International Conference on Learning Representations , year=

  40. [40]

    arXiv preprint arXiv:2401.10208 , year=

    Mm-interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer , author=. arXiv preprint arXiv:2401.10208 , year=

    Pith/arXiv arXiv

  41. [41]

    arXiv preprint arXiv:2405.09818 , year=

    Chameleon: Mixed-modal early-fusion foundation models , author=. arXiv preprint arXiv:2405.09818 , year=

    Pith/arXiv arXiv

  42. [42]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Generative multimodal models are in-context learners , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  43. [43]

    The Twelfth International Conference on Learning Representations , year=

    DreamLLM: Synergistic Multimodal Comprehension and Creation , author=. The Twelfth International Conference on Learning Representations , year=

  44. [44]

    arXiv preprint arXiv:2407.06135 , year=

    Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation , author=. arXiv preprint arXiv:2407.06135 , year=

    Pith/arXiv arXiv

  45. [45]

    European Conference on Computer Vision , pages=

    Learning Video Context as Interleaved Multimodal Sequences , author=. European Conference on Computer Vision , pages=

  46. [46]

    arXiv preprint arXiv:2508.21112 , year=

    Eo-1: Interleaved vision-text-action pretraining for general robot control , author=. arXiv preprint arXiv:2508.21112 , year=

    arXiv

  47. [47]

    arXiv preprint arXiv:2506.18701 , year=

    Matrix-Game: Interactive World Foundation Model , author=. arXiv preprint arXiv:2506.18701 , year=

    Pith/arXiv arXiv

  48. [48]

    arXiv preprint arXiv:2508.13009 , year=

    Matrix-game 2.0: An open-source real-time and streaming interactive world model , author=. arXiv preprint arXiv:2508.13009 , year=

    Pith/arXiv arXiv

  49. [49]

    Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

    STAR: A Benchmark for Situated Reasoning in Real-World Videos , author=. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

  50. [50]

    International Conference on Learning Representations , year=

    CLEVRER: Collision Events for Video Representation and Reasoning , author=. International Conference on Learning Representations , year=

  51. [51]

    Advances in Neural Information Processing Systems , volume=

    Perception test: A diagnostic benchmark for multimodal video models , author=. Advances in Neural Information Processing Systems , volume=

  52. [52]

    European Conference on Computer Vision , pages=

    Evaluating Text-to-Visual Generation with Image-to-Text Generation , author=. European Conference on Computer Vision , pages=

  53. [53]

    ACM Computing Surveys , volume=

    Understanding world or predicting future? a comprehensive survey of world models , author=. ACM Computing Surveys , volume=. 2025 , publisher=

    2025

  54. [54]

    arXiv preprint arXiv:1803.10122 , year=

    World Models , author=. arXiv preprint arXiv:1803.10122 , year=

    Pith/arXiv arXiv

  55. [55]

    The Twelfth International Conference on Learning Representations , year=

    Language Models Represent Space and Time , author=. The Twelfth International Conference on Learning Representations , year=

  56. [56]

    OpenReview , volume=

    A path towards autonomous machine intelligence , author=. OpenReview , volume=

  57. [57]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Reasoning with language model is planning with world model , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  58. [58]

    NeurIPS 2023 Workshop on Generalization in Planning , year=

    Learning Interactive Real-World Simulators , author=. NeurIPS 2023 Workshop on Generalization in Planning , year=

    2023

  59. [59]

    Sora: Creating video from text , author=

  60. [60]

    arXiv preprint arXiv:2402.17177 , year=

    Sora: A review on background, technology, limitations, and opportunities of large vision models , author=. arXiv preprint arXiv:2402.17177 , year=

    Pith/arXiv arXiv

  61. [61]

    Conference on robot learning , pages=

    Daydreamer: World models for physical robot learning , author=. Conference on robot learning , pages=. 2023 , organization=

    2023

  62. [62]

    Proceedings of the 36th annual acm symposium on user interface software and technology , pages=

    Generative agents: Interactive simulacra of human behavior , author=. Proceedings of the 36th annual acm symposium on user interface software and technology , pages=

  63. [63]

    Advances in Neural Information Processing Systems , volume=

    Vista: A generalizable driving world model with high fidelity and versatile controllability , author=. Advances in Neural Information Processing Systems , volume=

  64. [64]

    American Scientist , volume=

    Visual dynamics , author=. American Scientist , volume=. 1988 , publisher=

    1988

  65. [65]

    International Journal of Computer Vision , volume=

    Learning to Recognize Visual Dynamic Events from Examples , author=. International Journal of Computer Vision , volume=. 2000 , publisher=

    2000

  66. [66]

    Image and vision computing , volume=

    Learning and understanding dynamic scene activity: a review , author=. Image and vision computing , volume=. 2003 , publisher=

    2003

  67. [67]

    IEEE Computer Graphics and Applications , volume=

    Depicting dynamics using principles of visual art and narrations , author=. IEEE Computer Graphics and Applications , volume=. 2005 , publisher=

    2005

  68. [68]

    Advances in neural information processing systems , volume=

    Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks , author=. Advances in neural information processing systems , volume=

  69. [69]

    Advances in neural information processing systems , volume=

    Unsupervised learning for physical interaction through video prediction , author=. Advances in neural information processing systems , volume=

  70. [70]

    Proceedings of the National Academy of Sciences , volume=

    The emergence of visual objects in space--time , author=. Proceedings of the National Academy of Sciences , volume=. 2000 , publisher=

    2000

  71. [71]

    2007 , publisher=

    Dynamic vision for perception and control of motion , author=. 2007 , publisher=

    2007

  72. [72]

    Scientific American , volume=

    Visual motion perception , author=. Scientific American , volume=. 1975 , publisher=

    1975

  73. [73]

    Perception , volume=

    Visual representations of dynamic actions from static pictures , author=. Perception , volume=. 2005 , publisher=

    2005

  74. [74]

    Proceedings of the 6th British conference on Machine vision (Vol

    Statistical background modelling for tracking with a virtual camera , author=. Proceedings of the 6th British conference on Machine vision (Vol. 2) , pages=

  75. [75]

    arXiv preprint arXiv:2504.02004 , year=

    Beyond Static Scenes: Camera-controllable Background Generation for Human Motion , author=. arXiv preprint arXiv:2504.02004 , year=

    Pith/arXiv arXiv

  76. [76]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Mad: A scalable dataset for language grounding in videos from movie audio descriptions , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  77. [77]

    Advances in Neural Information Processing Systems , volume=

    Minedojo: Building open-ended embodied agents with internet-scale knowledge , author=. Advances in Neural Information Processing Systems , volume=

  78. [78]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Koala-36m: A large-scale video dataset improving consistency between fine-grained conditions and video content , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  79. [79]

    arXiv preprint arXiv:2508.09983 , year=

    Story2board: A training-free approach for expressive storyboard generation , author=. arXiv preprint arXiv:2508.09983 , year=

    Pith/arXiv arXiv

  80. [80]

    Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

    Synthesizing coherent story with auto-regressive latent diffusion models , author=. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

Showing first 80 references.