Pith. sign in

REVIEW 3 major objections 52 references

Different action interfaces can share one video world model when experts separate common world dynamics from control-specific details.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 22:41 UTC pith:ZITXFZHK

load-bearing objection Solid systems paper that unifies camera/robot/hand control in one DiT-MoE world model; the multi-benchmark wins are real, but the MoE-vs-dense ablation does not cleanly prove specialization over capacity. the 3 major comments →

arxiv 2607.03964 v1 pith:ZITXFZHK submitted 2026-07-04 cs.RO cs.AI

Worldscape-MoE: A Unified Mixture-of-Experts World Model for Scalable Heterogeneous Action Control

Jianjie Fang , Yongyan Xu , Ziyou Wang , Chen Gao , Yuchao Huang , Zhaolu Wang , Rongze Tang , Mingyuan Jia
show 8 more authors
Baining Zhao Weichen Zhang Xin Zhang Haisheng Su Yu Shang Wei Wu Xinlei Chen Yong Li
This is my paper
classification cs.RO cs.AI
keywords world modelsmixture-of-expertsdiffusion transformersheterogeneous action controlembodied AIvideo generationrobotic manipulationegocentric interaction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Video world models are still built as separate systems for separate controls—camera paths, robot joint commands, hand-joint maps—even though those signals all constrain the same scenes and physical regularities. This paper argues the scaling bottleneck is not missing controllable generators but the lack of one extensible framework that can absorb mixed action supervision without destroying a shared model of how the world evolves. Worldscape-MoE is a Diffusion Transformer that injects each control through a pathway matched to its structure, then replaces dense feed-forward layers with a control-aware Mixture-of-Experts block: every sample always updates a shared expert and only the experts for the controls it carries. Progressive tuning adds new experts by cloning the shared expert and training shared weights more carefully, so new modalities can be absorbed without rebuilding the whole model. Across locomotion, robotic manipulation, and egocentric hand control, joint heterogeneous training improves rather than interferes with each interface, with gains on standard benchmarks, out-of-distribution scenes, and early loco-manipulation composition.

Core claim

Heterogeneous action supervision improves individual control capabilities when a single Diffusion Transformer world model factors computation into a shared expert for cross-control physical regularities and modality-specific experts for each action interface, instead of forcing all controls through one dense pathway.

What carries the argument

Worldscape-MoE: modality-aware control injection into a shared DiT backbone plus control-aware MoE feed-forwards that always activate a shared expert and only eligible control-specific experts, trained with progressive expert expansion and grouped learning rates.

Load-bearing premise

The claim rests on the premise that masking experts by control type, always updating a shared expert at a lower learning rate, and cloning new experts from that shared expert is enough to separate common world physics from control-specific residuals—so measured gains are specialization, not just extra capacity or a lucky data mix.

What would settle it

Train a capacity-matched dense multi-branch baseline on the same mixed data, budget, and injection paths without modality-masked MoE routing; if it matches or beats Worldscape-MoE on locomotion, manipulation, and hand metrics while expert-loading no longer shows modality-dependent dedicated use, the specialization claim is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Fragmented camera, robot, and hand datasets can be pooled as one growing training resource instead of training siloed single-control models.
  • New action modalities can be added by extending experts and injection branches while retaining previously learned controls.
  • Shared experts transfer physical and scene regularities to out-of-distribution environments and to new single-arm manipulation settings with limited fine-tuning.
  • Once multiple controls share one world representation, the model can begin to compose previously isolated skills such as loco-manipulation.
  • Controllable world models can scale by absorbing heterogeneous supervision rather than by repeatedly partitioning capacity along control boundaries.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the gains are truly from factorization, equal-parameter dense multi-branch models should still underperform even when data mix and injection paths are held fixed—a stricter capacity control the paper only partly isolates.
  • The same shared-plus-specific expert pattern may transfer to other action-conditioned predictors that currently keep separate stacks per robot or sensor suite.
  • Progressive cloning of experts from a shared world prior could become a practical recipe for lifelong addition of embodiments without full retraining.
  • Reliable loco-manipulation composition would reduce reliance on hand-designed skill libraries when planning over mixed action types in one simulator.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper proposes Worldscape-MoE, a DiT-based video world model that unifies heterogeneous action controls (camera locomotion, dual-arm robot actions, and dense hand-joint action maps) via modality-aware injection, a control-aware MoE-FFN with a always-on shared expert plus eligibility-masked modality experts (Eqs. 4–7), and progressive Worldscape-MoE Tuning with shared-expert cloning and grouped learning rates. It argues that different controls are interfaces to shared world dynamics, so joint training should improve rather than interfere with individual controls (RQ1–RQ4). Empirically, the model reports leading aggregate scores on iWorld-Bench locomotion (Table 1), WorldArena EWMScore for dual-arm manipulation (Table 2; 62.84), and EgoDex-subset hand metrics (Table 3), with routing load analysis (Fig. 5), dense mixed-training ablations, modality-extension notes, OOD qualitative cases, and loco-manipulation composition demos.

Significance. If the factorization claim holds, the work is a useful systems contribution for embodied world models: it reframes fragmented control-specific generators as a single extensible training resource and shows competitive multi-control performance on standard-ish benchmarks (WorldArena, iWorld-Bench-style locomotion, hand FID/FVD). Strengths include a clear architectural recipe (asymmetric injection + shared/dedicated experts), progressive expansion protocol, multi-regime evaluation, and expert-loading diagnostics. The significance is primarily empirical and engineering rather than theoretical; the paper would matter most if it cleanly shows that MoE specialization—not just capacity, data mix, or staged multi-branch conditioning—is what turns heterogeneous supervision into a scaling resource.

major comments (3)
  1. The central factorization claim (shared expert = world dynamics; dedicated experts = control residuals; heterogeneous training improves rather than interferes) is under-identified by the decisive ablation. §3.4 and Tables 1–3 compare Worldscape-MoE to a dense “w/o MoE” mixed-training baseline under a claimed same budget, but MoE multiplies FFN capacity (Eq. 7), uses progressive expert cloning and grouped LRs (§2.4), and routes by deterministic modality eligibility. Without a capacity-matched multi-expert dense control, multi-branch dense conditioner, or parameter-matched shared-only MoE, gains may reflect extra parameters and better-conditioned adapters rather than the claimed shared/specific factorization (RQ2).
  2. Aggregate leadership overstates cross-control fidelity. In Table 1, Worldscape-MoE’s overall locomotion score (0.7556) leads, but Trajectory Accuracy (0.6300) trails VideoX-Fun-Wan (0.7645) and RealCam-I2V (0.7050); motion smoothness is near ceiling for many methods. The paper should report primary control-faithfulness metrics with equal weight to aggregates, and state clearly where unified training trades off pure trajectory following.
  3. RQ3 scalability and “improves rather than interferes” are only partially evidenced. §3.4 and Appendix D.3 describe transient locomotion degradation after adding a modality and later recovery, plus expert L2 update tables, but there is no quantitative before/after table of all primary metrics when experts/data are added, no multi-seed statistics, and no held-out joint-control quantitative protocol for loco-manipulation (Fig. 32 is qualitative). The continual-extension claim needs a compact scaling table and uncertainty estimates.

Circularity Check

0 steps flagged

No significant circularity: empirical MoE world-model claims are tested on held-out benchmarks, not derived by redefining targets as fitted inputs.

full rationale

Worldscape-MoE is an empirical systems/ML paper. Its load-bearing claims—that a DiT with shared plus modality-specific experts can absorb heterogeneous action supervision without cross-control interference, and that joint training improves locomotion, manipulation, and hand-control metrics—are evaluated by training and scoring on external or held-out protocols (iWorld-Bench locomotion set, WorldArena EWMScore, EgoDex hand subset), with ablations (w/o MoE dense mixed training, routing workload, progressive extension). There is no first-principles derivation in which a predicted quantity is algebraically identical to a fitted constant, no uniqueness theorem imported from the authors that forces the architecture, and no ansatz smuggled in as a mathematical necessity. Self-citations (e.g., WorldArena, iWorld-Bench, Matrix-Game) supply evaluation protocols and related-work context; they do not close a definitional loop with the reported scores. Design choices (deterministic modality eligibility, shared expert always updated, progressive expert cloning, grouped LRs) are architectural hypotheses tested empirically, not circular identities. Concerns that the MoE-vs-dense comparison under-isolates capacity versus specialization are experimental-identification issues, not circularity of the derivation chain.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

The central claim rests on standard diffusion-transformer video priors, the domain premise that heterogeneous controls share latent world dynamics, and several design choices (eligibility routing, grouped LRs, expert cloning, asymmetric injection) that are architectural axioms rather than derived necessities. Free parameters are training/architecture knobs; invented entities are the MoE factorization components themselves, evidenced only by ablations inside this paper.

free parameters (5)
  • Router temperature τ
    Softmax temperature in eligibility-masked routing (§2.3); controls sharpness of expert mixture weights.
  • Grouped learning rates (shared vs modality-specific)
    Shared expert and some adapters use a smaller LR than new experts/adapters (§2.4); chosen to preserve priors and enable specialization.
  • Progressive stage schedule and expert-init cloning
    Stage I camera+arm then Stage II hand (and further extensions); new experts initialized from current shared expert (§2.4)—hand-designed curriculum.
  • Action tensor shape Ta=17, dact=14
    Fixed dual-arm action window used as the manipulation interface (§2.1); defines the control representation.
  • Number of DiT MoE layers / expert set size M+1
    Architectural capacity allocation; not derived from theory.
axioms (5)
  • domain assumption Heterogeneous controls (camera, robot actions, hand maps) are different interfaces to one shared latent world with reusable physical and temporal regularities.
    Core design principle in abstract and §1–2.1; motivates shared expert rather than isolated models.
  • domain assumption A pretrained video DiT/VAE/text encoder provides transferable spatiotemporal priors that fine-tuning should preserve.
    Frozen VAE/text encoder and expert init from pretrained FFN (§2.3–2.4).
  • ad hoc to paper Deterministic modality eligibility (shared always on; modality expert only if control present) induces useful specialization without auxiliary load-balancing loss.
    Explicit design choice in §2.3; specialization attributed to eligibility + diffusion loss only.
  • ad hoc to paper Asymmetric injection—dense controls via visual tokens, compact actions via timestep modulation—is an appropriate unified conditioning form.
    Eq. (3) and §2.3; structural assumption of the architecture.
  • domain assumption Standard diffusion training and video-quality / EWM metrics are adequate proxies for world-model utility under action control.
    Evaluation protocol §3.1 and Appendix D.2; functional robot closed-loop utility is not the primary reported objective.
invented entities (3)
  • Worldscape-MoE control-aware MoE-FFN (shared + modality experts with eligibility-masked router) no independent evidence
    purpose: Factor shared world dynamics from control-specific action interpretation under mixed supervision.
    Primary architectural invention (§2.3); independent evidence limited to internal ablations and routing stats.
  • Worldscape-MoE Tuning (progressive control expansion with shared-expert cloning and grouped LRs) no independent evidence
    purpose: Continual extension to new action modalities without full relearning.
    Training procedure §2.4; support is extension experiments with noted transient degradation.
  • Modality-aware heterogeneous control injection pathways (Control Adapter / VAE action maps / Action MLP) no independent evidence
    purpose: Map structurally different action signals into one DiT backbone.
    §2.3 Eq. (3); engineering construct validated by multi-control generation quality.

pith-pipeline@v1.1.0-grok45 · 27859 in / 3713 out tokens · 36804 ms · 2026-07-11T22:41:42.446051+00:00 · methodology

0 comments
read the original abstract

World models are rapidly becoming a core infrastructure for embodied intelligence and interactive agents: they provide controllable simulators in which agents can perceive, act, forecast, and acquire scalable experience. Yet current video generation world models are still organized around isolated control interfaces, such as camera trajectories, robot actions, or hand-joint signals. This fragmentation is increasingly a scaling bottleneck. The central challenge is not the absence of controllable generators, but the lack of a unified and extensible learning framework that can absorb heterogeneous action supervision while preserving a shared model of world dynamics. In this work, we introduce Worldscape-MoE, a Mixture-of-Experts world model built on Diffusion Transformers for scalable heterogeneous action control. Our key observation is that different controls specify different interfaces to the same underlying world: although their representations differ, they constrain shared physical regularities, scene dynamics, and interaction semantics. Worldscape-MoE operationalizes this observation through modality-aware control injection, shared and control-specific experts, and a progressive MoE tuning strategy that supports continual extension to new action modalities. Experiments across locomotion, robotic manipulation, and egocentric hand control show that heterogeneous supervision improves rather than interferes with individual control capabilities. Worldscape-MoE achieves strong results on WorldArena, improves locomotion and hand-control metrics, exhibits robust out-of-distribution generalization, and demonstrates scaling behavior as additional control data and experts are integrated.

Figures

Figures reproduced from arXiv: 2607.03964 by Baining Zhao, Chen Gao, Haisheng Su, Jianjie Fang, Mingyuan Jia, Rongze Tang, Weichen Zhang, Wei Wu, Xinlei Chen, Xin Zhang, Yong Li, Yongyan Xu, Yuchao Huang, Yu Shang, Zhaolu Wang, Ziyou Wang.

Figure 1
Figure 1. Figure 1: Worldscape-MoE Overview. Worldscape-MoE supports three mainstream control modalities: Locomotion for trajectory-conditioned world navigation, Manipulation for robot-action-conditioned embodied tasks, and Action Map for hand-joint-conditioned egocentric interaction generation. The framework can also be extended to additional control injection settings. as Ctrl-World [12] and Cosmos-Predict [13], emphasize r… view at source ↗
Figure 2
Figure 2. Figure 2: Worldscape-MoE Architecture. Given the current world observation and different forms of supervisory control, our framework generates world dynamics under heterogeneous control signals. It supports both egocentric world exploration and embodied task execution. additional visual conditions for the diffusion backbone. Manipulation, in contrast, is represented as a low-dimensional action sequence that specifie… view at source ↗
Figure 3
Figure 3. Figure 3: Worldscape-MoE Tuning. The shared expert is optimized with a smaller learning rate, while each modality-specific expert uses a larger learning rate for control specialization. Progressive Control Expansion. To stably extend the model to multiple control capabilities, we adopt a progressive modality-incremental strategy rather than introducing all experts and control branches at once. The process consists o… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison under heterogeneous controls. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Layer-wise MoE expert loading analysis. Gate-weighted expert workload across MoE layers. The panels show all-expert loading, dedicated-expert loading, and shared-vs-dedicated routing for locomotion, manipulation, and hand-motion control. 3.4 MoE Ablation and Modality Extension Effectiveness of MoE. To isolate the contribution of the MoE design, we compare Worldscape-MoE with a direct mixed-training baselin… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative evidence for the effectiveness of MoE. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Out-of-distribution detection across three motion domains. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Seedance 2.0 Data Augmentation [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Transient Degradation and Recovery During Modality Extension. [PITH_FULL_IMAGE:figures/full_fig_p025_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Single-arm LIBERO generalization. Each panel compares the ground-truth demonstration with the model initialized from the MoE shared expert after 5K fine-tuning steps. The figure is pre-rendered from normalized comparison videos by uniformly sampling seven frames from each rollout. D.5 More Qualitative Results of Worldscape-MoE In this section, we present more qualitative results of Worldscape-MoE [PITH_F… view at source ↗
Figure 11
Figure 11. Figure 11: Locomotion comparison. Each row compares video frames generated by Worldscape-MoE, CamI2V, Matrixgame, and RealCam-I2V under the same locomotion trajectory, with columns uniformly sampled from each rollout. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Locomotion comparison. Each row compares video frames generated by Worldscape-MoE, Astra, Matrixgame, and RealCam-I2V under the same locomotion trajectory, with columns uniformly sampled from each rollout. L o c o m o t i o n M otio n Ctrl Astr a M atrix g a m e Wo rldsc a p e -M o E [PITH_FULL_IMAGE:figures/full_fig_p028_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Locomotion comparison. Each row compares video frames generated by Worldscape-MoE, Astra, Matrixgame, and MotionCtrl under the same locomotion trajectory, with columns uniformly sampled from each rollout. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Locomotion comparison. Each row compares video frames generated by Worldscape-MoE, VideoX￾fun-Wan, Matrixgame, and CameraCtrl under the same locomotion trajectory, with columns uniformly sampled from each rollout. L o c o m o t i o n vid e o x￾fu n-W a n C a mI2 V M atrix g a m e Wo rldsc a p e -M o E [PITH_FULL_IMAGE:figures/full_fig_p029_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Locomotion comparison. Each row compares video frames generated by Worldscape-MoE, CamI2V, Matrixgame, and VideoX-fun-Wan under the same locomotion trajectory, with columns uniformly sampled from each rollout. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Locomotion comparison. Each row compares video frames generated by Worldscape-MoE, CamI2V, HY-World, and CameraCtrl under the same locomotion trajectory, with columns uniformly sampled from each rollout [PITH_FULL_IMAGE:figures/full_fig_p030_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Manipulation comparison. Each row compares video frames generated by Worldscape-MoE, IRASim, Cosmos-predict2.5, and CtrlWorld under the same manipulation trajectory, with columns uniformly sampled from each rollout. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Manipulation comparison. Each row compares video frames generated by Worldscape-MoE, IRASim, Cosmos-predict2.5, and CtrlWorld under the same manipulation trajectory, with columns uniformly sampled from each rollout [PITH_FULL_IMAGE:figures/full_fig_p031_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Manipulation comparison. Each row compares video frames generated by Worldscape-MoE, IRASim, Cosmos-predict2.5, and CtrlWorld under the same manipulation trajectory, with columns uniformly sampled from each rollout. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Manipulation comparison. Each row compares video frames generated by Worldscape-MoE, IRASim, Cosmos-predict2.5, and CtrlWorld under the same manipulation trajectory, with columns uniformly sampled from each rollout [PITH_FULL_IMAGE:figures/full_fig_p032_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Manipulation comparison. Each row compares video frames generated by Worldscape-MoE, IRASim, Cosmos-predict2.5, and CtrlWorld under the same manipulation trajectory, with columns uniformly sampled from each rollout. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Manipulation comparison. Each row compares video frames generated by Worldscape-MoE, IRASim, Cosmos-predict2.5, and CtrlWorld under the same manipulation trajectory, with columns uniformly sampled from each rollout [PITH_FULL_IMAGE:figures/full_fig_p033_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Hand motion comparison. Each row compares video frames generated by Worldscape-MoE, MimicMotion, and HunyuanVideo under the same hand motion trajectory, with columns uniformly sampled from each rollout [PITH_FULL_IMAGE:figures/full_fig_p033_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Hand motion comparison. Each row compares video frames generated by Worldscape-MoE, MimicMotion, and HunyuanVideo under the same hand motion trajectory, with columns uniformly sampled from each rollout. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Hand motion comparison. Each row compares video frames generated by Worldscape-MoE, MimicMotion, and HunyuanVideo under the same hand motion trajectory, with columns uniformly sampled from each rollout [PITH_FULL_IMAGE:figures/full_fig_p034_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Hand motion comparison. Each row compares video frames generated by Worldscape-MoE, MimicMotion, and HunyuanVideo under the same hand motion trajectory, with columns uniformly sampled from each rollout [PITH_FULL_IMAGE:figures/full_fig_p034_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Hand motion comparison. Each row compares video frames generated by Worldscape-MoE, MimicMotion, and HunyuanVideo under the same hand motion trajectory, with columns uniformly sampled from each rollout [PITH_FULL_IMAGE:figures/full_fig_p034_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Hand motion comparison. Each row compares video frames generated by Worldscape-MoE, MimicMotion, and HunyuanVideo under the same hand motion trajectory, with columns uniformly sampled from each rollout. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: Hand motion comparison. Each row compares video frames generated by Worldscape-MoE, MimicMotion, and HunyuanVideo under the same hand motion trajectory, with columns uniformly sampled from each rollout. With o ut M o E L o c o m o tio n With M o E With o ut M o E M a nip ula tio n With M o E With o ut M o E H a n d m o tio n With M o E [PITH_FULL_IMAGE:figures/full_fig_p035_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: Effect of MoE across diverse motion scenarios. [PITH_FULL_IMAGE:figures/full_fig_p035_30.png] view at source ↗
Figure 31
Figure 31. Figure 31: Physics consistency across diverse visual domains. [PITH_FULL_IMAGE:figures/full_fig_p035_31.png] view at source ↗
Figure 32
Figure 32. Figure 32: Loco-manipulation generation across diverse scenes. [PITH_FULL_IMAGE:figures/full_fig_p036_32.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 28 linked inside Pith

  1. [1]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Hunyuanvideo 1.5 technical report.arXiv preprint arXiv:2511.18870, 2025

    Bing Wu, Chang Zou, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Jack Peng, Jianbing 13 Wu, Jiangfeng Xiong, Jie Jiang, et al. Hunyuanvideo 1.5 technical report.arXiv preprint arXiv:2511.18870, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Understanding world or predicting future? a comprehensive survey of world models.ACM Computing Surveys, 58(3):1–38, 2025

    Jingtao Ding, Yunke Zhang, Yu Shang, Yuheng Zhang, Zefang Zong, Jie Feng, Yuan Yuan, Hongyuan Su, Nian Li, Nicholas Sukiennik, et al. Understanding world or predicting future? a comprehensive survey of world models.ACM Computing Surveys, 58(3):1–38, 2025

    2025

  5. [5]

    Gigaworld-0: World models as data engine to empower embodied ai.arXiv preprint arXiv:2511.19861, 2025

    GigaWorld Team, Angen Ye, Boyuan Wang, Chaojun Ni, Guan Huang, Guosheng Zhao, Haoyun Li, Jiagang Zhu, Kerui Li, Mengyuan Xu, et al. Gigaworld-0: World models as data engine to empower embodied ai.arXiv preprint arXiv:2511.19861, 2025

    arXiv 2025

  6. [6]

    Genie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. InForty-first International Conference on Machine Learning, 2024

    2024

  7. [7]

    Matrix-game 2.0: An open-source real-time and streaming interactive world model.arXiv preprint arXiv:2508.13009, 2025

    Xianglong He, Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Biao Jiang, Mengyin An, Yangyang Ren, et al. Matrix-game 2.0: An open-source real-time and streaming interactive world model.arXiv preprint arXiv:2508.13009, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Matrix-game 3.0: Real-time and streaming interactive world model with long-horizon memory.arXiv preprint arXiv:2604.08995, 2026

    Zile Wang, Zexiang Liu, Jaixing Li, Kaichen Huang, Baixin Xu, Fei Kang, Mengyin An, Peiyu Wang, Biao Jiang, Yichen Wei, et al. Matrix-game 3.0: Real-time and streaming interactive world model with long-horizon memory.arXiv preprint arXiv:2604.08995, 2026

    Pith/arXiv arXiv 2026

  9. [9]

    Matrix-game: Interactive world foundation model.arXiv preprint arXiv:2506.18701, 2025

    Yifan Zhang, Chunli Peng, Boyang Wang, Puyi Wang, Qingcheng Zhu, Fei Kang, Biao Jiang, Zedong Gao, Eric Li, Yang Liu, et al. Matrix-game: Interactive world foundation model.arXiv preprint arXiv:2506.18701, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Worldplay: Towards long-term geometric consistency for real-time interactive world modeling.arXiv preprint arXiv:2512.14614, 2025

    Wenqiang Sun, Haiyu Zhang, Haoyuan Wang, Junta Wu, Zehan Wang, Zhenwei Wang, Yunhong Wang, Jun Zhang, Tengfei Wang, and Chunchao Guo. Worldplay: Towards long-term geometric consistency for real-time interactive world modeling.arXiv preprint arXiv:2512.14614, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Hy-world 2.0: A multi-modal world model for reconstructing, generating, and simulating 3d worlds.arXiv preprint arXiv:2604.14268, 2026

    Team HY-World, Chenjie Cao, Xuhui Zuo, Zhenwei Wang, Yisu Zhang, Junta Wu, Zhenyang Liu, Yuning Gong, Yang Liu, Bo Yuan, et al. Hy-world 2.0: A multi-modal world model for reconstructing, generating, and simulating 3d worlds.arXiv preprint arXiv:2604.14268, 2026

    Pith/arXiv arXiv 2026

  12. [12]

    Ctrl-world: A controllable generative world model for robot manipulation.arXiv preprint arXiv:2510.10125, 2025

    Yanjiang Guo, Lucy Xiaoyang Shi, Jianyu Chen, and Chelsea Finn. Ctrl-world: A controllable generative world model for robot manipulation.arXiv preprint arXiv:2510.10125, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    World simulation with video foundation models for physical ai.arXiv preprint arXiv:2511.00062, 2025

    Arslan Ali, Junjie Bai, Maciej Bala, Yogesh Balaji, Aaron Blakeman, Tiffany Cai, Jiaxin Cao, Tianshi Cao, Elizabeth Cha, Yu-Wei Chao, et al. World simulation with video foundation models for physical ai.arXiv preprint arXiv:2511.00062, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Worldarena: A unified benchmark for evaluating perception and functional utility of embodied world models

    Yu Shang, Zhuohang Li, Yiding Ma, Weikang Su, Xin Jin, Yinzhou Tang, Haisheng Su, Chen Gao, Wei Wu, Xihui Liu, Zhibo Chen, Jun Zhu, Yonghong Tian, Tat-Seng Chua, Ziyou Wang, Dhruv Shah, Wenwu Zhu, Lei Jin, Xin Zhang, Zhaoxiang Zhang, and Yong Li. Worldarena: A unified benchmark for evaluating perception and functional utility of embodied world models. arX...

    arXiv 2026

  15. [15]

    Hand2world: Autoregressive egocentric interaction generation via free-space hand gestures.arXiv preprint arXiv:2602.09600, 2026

    Yuxi Wang, Wenqi Ouyang, Tianyi Wei, Yi Dong, Zhiqi Shen, and Xingang Pan. Hand2world: Autoregressive egocentric interaction generation via free-space hand gestures.arXiv preprint arXiv:2602.09600, 2026. 14

    arXiv 2026

  16. [16]

    Generated reality: Human-centric world simulation using interactive video generation with hand and camera control.arXiv preprint arXiv:2602.18422, 2026

    Linxi Xie, Lisong C Sun, Ashley Neall, Tong Wu, Shengqu Cai, and Gordon Wetzstein. Generated reality: Human-centric world simulation using interactive video generation with hand and camera control.arXiv preprint arXiv:2602.18422, 2026

    arXiv 2026

  17. [17]

    Stereo magni- fication: Learning view synthesis using multiplane images.arXiv preprint arXiv:1805.09817, 2018

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magni- fication: Learning view synthesis using multiplane images.arXiv preprint arXiv:1805.09817, 2018

    Pith/arXiv arXiv 2018

  18. [18]

    Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision

    Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22160–22169, 2024

    2024

  19. [19]

    iworld-bench: A benchmark for interactive world models with a unified action generation framework.arXiv preprint arXiv:2605.03941, 2026

    Jianjie Fang, Yingshan Lei, Qin Wan, Ziyou Wang, Yuchao Huang, Yongyan Xu, Baining Zhao, Weichen Zhang, Chen Gao, Xinlei Chen, and Yong Li. iworld-bench: A benchmark for interactive world models with a unified action generation framework.arXiv preprint arXiv:2605.03941, 2026

    Pith/arXiv arXiv 2026

  20. [20]

    Sekai: A video dataset towards world exploration.arXiv preprint arXiv:2506.15675, 2025

    Zhen Li, Chuanhao Li, Xiaofeng Mao, Shaoheng Lin, Ming Li, Shitian Zhao, Zhaopan Xu, Xinyue Li, Yukang Feng, Jianwen Sun, et al. Sekai: A video dataset towards world exploration.arXiv preprint arXiv:2506.15675, 2025

    arXiv 2025

  21. [21]

    Spatialvid: A large-scale video dataset with spatial annotations.arXiv preprint arXiv:2509.09676, 2025

    Jiahao Wang, Yufeng Yuan, Rujie Zheng, Youtian Lin, Jian Gao, Lin-Zhuo Chen, Yajie Bao, Yi Zhang, Chang Zeng, Yanxi Zhou, et al. Spatialvid: A large-scale video dataset with spatial annotations.arXiv preprint arXiv:2509.09676, 2025

    arXiv 2025

  22. [22]

    Vipe: Video pose engine for 3d geometric perception.arXiv preprint arXiv:2508.10934, 2025

    Jiahui Huang, Qunjie Zhou, Hesam Rabeti, Aleksandr Korovko, Huan Ling, Xuanchi Ren, Tianchang Shen, Jun Gao, Dmitry Slepichev, Chen-Hsuan Lin, et al. Vipe: Video pose engine for 3d geometric perception.arXiv preprint arXiv:2508.10934, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Robotwin: Dual-arm robot benchmark with generative digital twins

    Yao Mu, Tianxing Chen, Zanxin Chen, Shijia Peng, Zhiqian Lan, Zeyu Gao, Zhixuan Liang, Qiaojun Yu, Yude Zou, Mingkun Xu, et al. Robotwin: Dual-arm robot benchmark with generative digital twins. InProceedings of the computer vision and pattern recognition conference, pages 27649–27660, 2025

    2025

  24. [24]

    Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation.arXiv preprint arXiv:2506.18088, 2025

    Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Zixuan Li, Qiwei Liang, Xianliang Lin, Yiheng Ge, Zhenyu Gu, et al. Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation.arXiv preprint arXiv:2506.18088, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Seedance 2.0: Advancing video generation for world complexity.arXiv preprint arXiv:2604.14148, 2026

    Team Seedance, De Chen, Liyang Chen, Xin Chen, Ying Chen, Zhuo Chen, Zhuowei Chen, Feng Cheng, Tianheng Cheng, Yufeng Cheng, et al. Seedance 2.0: Advancing video generation for world complexity.arXiv preprint arXiv:2604.14148, 2026

    Pith/arXiv arXiv 2026

  26. [26]

    Qwen3.6-Plus: Towards real world agents, April 2026

    Qwen Team. Qwen3.6-Plus: Towards real world agents, April 2026. URLhttps://qwen.ai/ blog?id=qwen3.6

    2026

  27. [27]

    Libero: Benchmarking knowledge transfer for lifelong robot learning.arXiv preprint arXiv:2306.03310, 2023

    Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning.arXiv preprint arXiv:2306.03310, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Egodex: Learning dexterous manipulation from large-scale egocentric video.arXiv preprint arXiv:2505.11709, 2025

    Ryan Hoque, Peide Huang, David J Yoon, Mouli Sivapurapu, and Jian Zhang. Egodex: Learning dexterous manipulation from large-scale egocentric video.arXiv preprint arXiv:2505.11709, 2025. 15

    Pith/arXiv arXiv 2025

  29. [29]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18995–19012, 2022

    2022

  30. [30]

    Reconstructing hands in 3d with transformers

    Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9826–9836, 2024

    2024

  31. [31]

    Cosmos-transfer1: Conditional world generation with adaptive multimodal control.arXiv preprint arXiv:2503.14492, 2025

    Hassan Abu Alhaija, Jose Alvarez, Maciej Bala, Tiffany Cai, Tianshi Cao, Liz Cha, Joshua Chen, Mike Chen, Francesco Ferroni, Sanja Fidler, et al. Cosmos-transfer1: Conditional world generation with adaptive multimodal control.arXiv preprint arXiv:2503.14492, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Cameractrl: Enabling camera control for video diffusion models

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for video diffusion models. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  33. [33]

    Motionctrl: A unified and flexible motion controller for video generation

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024

    2024

  34. [34]

    Cami2v: Camera-controlled image-to-video diffusion model.arXiv preprint arXiv:2410.15957, 2024

    Guangcong Zheng, Teng Li, Rui Jiang, Yehao Lu, Tao Wu, and Xi Li. Cami2v: Camera-controlled image-to-video diffusion model.arXiv preprint arXiv:2410.15957, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Realcam-i2v: Real-world image-to-video generation with interactive complex camera control

    Teng Li, Guangcong Zheng, Rui Jiang, Shuigen Zhan, Tao Wu, Yehao Lu, Yining Lin, Chuanyun Deng, Yepan Xiong, Min Chen, et al. Realcam-i2v: Real-world image-to-video generation with interactive complex camera control. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 28785–28796, 2025

    2025

  36. [36]

    Videox-fun: A flexible framework for video generation at any resolution

    AIGC-Apps and Alibaba PAI Team. Videox-fun: A flexible framework for video generation at any resolution. https://github.com/aigc-apps/VideoX-Fun, 2024. Open-source project / technical framework

    2024

  37. [37]

    Ac3d: Analyzing and improving 3d camera control in video diffusion transformers

    Sherwin Bahmani, Ivan Skorokhodov, Guocheng Qian, Aliaksandr Siarohin, Willi Menapace, Andrea Tagliasacchi, David B Lindell, and Sergey Tulyakov. Ac3d: Analyzing and improving 3d camera control in video diffusion transformers. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 22875–22889, 2025

    2025

  38. [38]

    Mimicmotion: High-quality human motion video generation with confidence-aware pose guidance.arXiv preprint arXiv:2406.19680, 2024

    Yuang Zhang, Jiaxi Gu, Li-Wen Wang, Han Wang, Junqi Cheng, Yuefeng Zhu, and Fangyuan Zou. Mimicmotion: High-quality human motion video generation with confidence-aware pose guidance.arXiv preprint arXiv:2406.19680, 2024

    Pith/arXiv arXiv 2024

  39. [39]

    Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion.arXiv preprint arXiv:2311.12052, 2023

    Di Chang, Yichun Shi, Quankai Gao, Jessica Fu, Hongyi Xu, Guoxian Song, Qing Yan, Yizhe Zhu, Xiao Yang, and Mohammad Soleymani. Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion.arXiv preprint arXiv:2311.12052, 2023

    Pith/arXiv arXiv 2023

  40. [40]

    Lome: Learninghuman-object manipulation with action-conditioned egocentric world model.arXiv preprint arXiv:2603.27449, 2026

    QuankaiGao, JiaweiYang, QiangengXu, LeChen, andYueWang. Lome: Learninghuman-object manipulation with action-conditioned egocentric world model.arXiv preprint arXiv:2603.27449, 2026

    arXiv 2026

  41. [41]

    Adaptive mixtures of local experts.Neural computation, 3(1):79–87, 1991

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts.Neural computation, 3(1):79–87, 1991. 16

    1991

  42. [42]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. InInternational Conference on Learning Representations, 2017. doi: 1701.06538. URL https://arxiv.org/abs/1701.06538

    Pith/arXiv arXiv 2017

  43. [43]

    Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020

    Pith/arXiv arXiv 2006

  44. [44]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.arXiv preprint arXiv:2101.03961, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.arXiv preprint arXiv:2101.03961, 2022

    Pith/arXiv arXiv 2022

  45. [45]

    Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.arXiv preprint arXiv:2401.06066, 2024

    Pith/arXiv arXiv 2024

  46. [46]

    Moe-llava: Mixture of experts for large vision-language models.arXiv preprint arXiv:2401.15947, 2024

    Bin Lin, Zhenyu Tang, Yang Ye, Jinfa Huang, Junwu Zhang, Yatian Pang, Peng Jin, Munan Ning, Jiebo Luo, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models.arXiv preprint arXiv:2401.15947, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Deepseek-vl2: Mixture-of-experts vision- language models for advanced multimodal understanding.arXiv preprint arXiv:2412.10302, 2024

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. Deepseek-vl2: Mixture-of-experts visio...

    Pith/arXiv arXiv 2024

  48. [48]

    Nucleus-image: Sparse moe for image generation.arXiv preprint arXiv:2604.12163, 2026

    Chandan Akiti, Ajay Modukuri, Murali Nandan Nagarapu, Gunavardhan Akiti, and Haozhe Liu. Nucleus-image: Sparse moe for image generation.arXiv preprint arXiv:2604.12163, 2026

    Pith/arXiv arXiv 2026

  49. [49]

    Astra: General interactive world model with autoregressive denoising.arXiv preprint arXiv:2512.08931, 2026

    Yixuan Zhu, Jiaqi Feng, Wenzhao Zheng, Yuan Gao, Xin Tao, Pengfei Wan, Jie Zhou, and Jiwen Lu. Astra: General interactive world model with autoregressive denoising.arXiv preprint arXiv:2512.08931, 2026. URLhttps://arxiv.org/abs/2512.08931

    arXiv 2026

  50. [50]

    Wan-animate: Unified character animation and replacement with holistic replication.arXiv preprint arXiv:2509.14055, 2025

    Gang Cheng, Xin Gao, Li Hu, Siqi Hu, Mingyang Huang, Chaonan Ji, Ju Li, Dechao Meng, Jinwei Qi, Penchong Qiao, Zhen Shen, Yafei Song, Ke Sun, Linrui Tian, Feng Wang, Guangyuan Wang, Qi Wang, Zhongjian Wang, Jiayu Xiao, Sheng Xu, Bang Zhang, Peng Zhang, Xindi Zhang, Zhe Zhang, Jingren Zhou, and Lian Zhuo. Wan-animate: Unified character animation and replac...

    arXiv 2025

  51. [51]

    Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together.ACM Transactions on Graphics, 36(6), 2017. 17 A Broader Impacts This work may have several positive broader impacts. By unifying heterogeneous control modalities within a single world model, Worldscape-MoE provides a scalable framework ...

    2017

  52. [52]

    It uses a large-scale Vision 21 Transformer backbone and a Transformer decoder to regress hand parameters from a monocular RGB image

    is a fully Transformer-based method for hand mesh recovery. It uses a large-scale Vision 21 Transformer backbone and a Transformer decoder to regress hand parameters from a monocular RGB image. The method adopts the MANO parametric hand model [51] to represent hand geometry. Given a hand crop𝐼𝑡, HaMeR predicts the MANO pose parameters, shape parameters, a...

    arXiv 2083