The reviewed record of science sign in
Pith

arxiv: 2607.06559 · v1 · pith:OVCE752N · submitted 2026-07-07 · cs.RO

RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation

Haoyu Zhao , Xingyue Zhao , Siteng Huang , Xin Li , Deli Zhao , Zhongyu Li This is my paper

Reviewed by Pith2026-07-08 01:45 UTCglm-5.2pith:OVCE752Nopen to challenge →

classification cs.RO
keywords manipulationroboticrynnworld-4dworlddepthflowopticalactions
0
0 comments X

The pith

4D world model beats 2D policies on bimanual robot tasks

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that jointly generating RGB, depth, and optical flow within a single diffusion process produces a 4D scene representation that is structurally closer to robot end-effector actions than 2D pixel video alone. The authors build RynnWorld-4D, a tri-branch video diffusion transformer that co-produces future RGB frames, depth maps, and optical flow from one RGB-D image and a language instruction, using cross-modal attention and 3D rotary position embeddings to keep appearance, geometry, and motion mutually consistent. They then attach an inverse dynamics head—RynnWorld-4D-Policy—that reads the frozen world model's internal 4D latents in a single forward pass and outputs 54-dimensional bimanual action chunks at roughly 9 Hz, bypassing multi-step denoising at inference. The central claim is that this predictive 4D representation space, which makes per-pixel 3D structure and inter-frame 3D scene flow explicit, narrows the gap between visual prediction and low-level control, yielding state-of-the-art success rates on six real-world dexterous bimanual tasks and most notably outperforming foundation policies like π0.5 on tasks demanding spatial precision and temporal coordination such as hand-over, lid placement, and bowl stacking.

Core claim

The paper's central discovery is that a frozen diffusion world model trained to jointly predict RGB, depth, and optical flow contains internal latent features that, when consumed directly by a lightweight inverse dynamics head in a single forward pass, outperform both standard 2D video policies and large vision-language-action foundation models on precision-critical bimanual manipulation. The mechanism is that depth and optical flow, when back-projected under pinhole-camera geometry, yield per-point 3D scene flow, making the world model's latent space carry explicit kinetic and geometric cues that 2D pixel latents must instead infer implicitly from appearance residuals.

What carries the argument

The RGB-DF tri-branch diffusion transformer with Joint Cross-Modal Attention and 3D RoPE, trained in three stages (independent modality adaptation, frozen-backbone joint attention, full fine-tuning) on 254.4 million pseudo-annotated frames, serves as a frozen 4D visual encoder. A Flow Former compresses its concatenated tri-branch hidden states into spatio-temporal tokens, and a 4-step flow-matching inverse dynamics head decodes 10-action chunks from these tokens.

If this is right

  • If predictive 4D latents are indeed closer to action space than 2D latents, future robot policies could standardize on 4D world-model encoders rather than 2D vision encoders, trading compute for geometric grounding.
  • The single-forward-pass extraction strategy suggests that expensive diffusion denoising is needed only during world-model training, not during closed-loop control, potentially resolving the latency bottleneck that has kept generative video models out of real-time robotics.
  • The 254M-frame Rynn4DDataset 1.0 with pseudo-labeled depth and flow could become a general training resource for other 4D embodied models beyond this specific architecture.
  • The tri-branch cross-modal attention design with shared K/V and per-branch FFNs may offer a template for fusing additional modalities (e.g., surface normals, tactile signals) into a single diffusion loop.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The ablation comparing the full 4D encoder against a ResNet-18 baseline does not fully isolate whether the performance gain comes from 4D geometric grounding or from using a much larger pretrained transformer backbone. A cleaner test would compare the tri-branch 4D encoder against a single-branch transformer of equivalent parameter count and pretraining, which the paper does not provide.
  • The 9 Hz effective control frequency relies on action chunking (10 open-loop actions per 1.1 s planning cycle). If tasks require faster reactive correction within that 1.1 s window—such as catching a slipping object—the 9 Hz replan rate may be insufficient, and the paper's claim of robustness to mid-window perturbations is supported only qualitatively.
  • The pseudo-labels for depth and optical flow are produced by off-the-shelf foundation models (Depth Anything 3, DPFlow) trained on non-robotic data. Systematic biases in these pseudo-labels could propagate into the world model's 4D latents, and the paper does not measure sensitivity to pseudo-label quality.
  • If the core hypothesis—that explicit 3D scene flow latents are what drive the policy gains—holds, then adding a scene-flow consistency loss directly into the policy training objective (rather than relying on the frozen encoder to provide it implicitly) should further improve performance, which the paper does not test.

Load-bearing premise

The policy's performance gains are attributed to the 4D geometric and kinetic cues in the world model's latents, but the key ablation replaces the entire 4D encoder with a much smaller ResNet-18, making it impossible to distinguish whether the gains come from 4D structure or simply from using a vastly larger, pretrained transformer as the visual backbone.

What would settle it

A single-branch RGB-only transformer with the same parameter count, the same pretraining data, and the same inverse dynamics head would match or approach RynnWorld-4D-Policy's success rates, demonstrating that model scale and pretraining—not 4D geometric grounding—are the causal factors.

read the original abstract

Robotic manipulation in the open world requires not only recognizing what a scene looks like, but also anticipating how its 3D structure moves under interaction. We argue that synchronized RGB, depth, and optical flow, namely RGB-DF, provide a physically grounded representation that captures the underlying 4D dynamics of a scene. Compared to 2D pixel videos, this multi-modal synergy aligns visual appearance with geometric structure and temporal motion, creating a representation space significantly closer to the low-level end-effector actions demanded by robotic systems, thereby narrowing the gap between world prediction and policy learning. Building on this insight, we introduce RynnWorld-4D, a generative model that co-produces future RGB frames, depth maps, and optical flow from a single RGB-D image and a language instruction within one unified diffusion process. This 4D world model features a tri-branch architecture that integrates cross-modal attention with frame-wise 3D RoPE, ensuring that appearance, geometry, and motion evolve consistently. To supply training data at scale, we curate Rynn4DDataset 1.0, a massive dataset of over 254.4 million frames across egocentric human and robotic manipulation videos with high-quality pseudo-labels for depth and optical flow. We further propose RynnWorld-4D-Policy, an inverse dynamics head that consumes the internal 4D representations of RynnWorld-4D in a single forward pass, bypassing expensive multi-step denoising, to output robot actions in a closed-loop manner. Experiments show that RynnWorld-4D produces temporally and spatially coherent 4D predictions, and that RynnWorld-4D-Policy achieves state-of-the-art performance on real-world dexterous bimanual manipulation tasks, particularly excelling in tasks demanding spatial precision and temporal coordination.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 5 minor

Summary. This paper introduces RynnWorld-4D, a 4D embodied world model that co-generates synchronized RGB, depth, and optical flow (RGB-DF) sequences from a single RGB-D image and text instruction using a tri-branch diffusion transformer. The authors also curate Rynn4DDataset 1.0 (254.4M frames) with pseudo-labels for depth and flow, and propose RynnWorld-4D-Policy, an inverse dynamics head that consumes the frozen world model's internal 4D latents in a single forward pass to generate robotic actions. The system is evaluated on a 54-DOF bimanual dexterous manipulation platform (TIANJI M6 + WUJI Hand) across six real-world tasks, demonstrating strong performance in spatially precise tasks.

Significance. The paper presents a well-executed, large-scale effort integrating generative world modeling with robotic control. The tri-branch architecture with Joint Cross-Modal Attention and the phased training strategy (modality adaptation, frozen-backbone joint attention, full SFT) is technically sound. The provision of a large-scale dataset, reproducible code, and model weights is a notable strength. The core hypothesis—that jointly modeling RGB, depth, and flow within one diffusion loop provides predictive 4D features closer to robot actions than 2D pixel latents—is well-motivated and supported by the within-backbone ablation (Table 5). The system demonstrates impressive real-world dexterous manipulation results.

major comments (3)
  1. §4.3, Table 5: The headline claim of 'state-of-the-art performance' against foundation models π0 and π0.5 is potentially confounded by an embodiment mismatch. The paper evaluates these models on a 54-DOF dexterous hand system (TIANJI/WUJI), but never specifies whether π0/π0.5 were fine-tuned on the 200 episodes/task used for RynnWorld-4D-Policy or run zero-shot. If zero-shot, the low scores (e.g., π0.5 achieving 0% on Hand-over) reflect embodiment mismatch rather than a limitation of 2D representations. The paper must explicitly state the training protocol for these baselines to ensure data and compute are controlled. Without this, the SOTA claim is unsubstantiated.
  2. §4.3, Table 5: The 'w/o RynnWorld-4D' ablation uses a ResNet-18 backbone, which is drastically weaker in scale and pretraining than the proposed 5B-parameter diffusion transformer. This makes it impossible to isolate whether the 4D representation or the model scale drives the gains. While the within-backbone ablations (RGB vs. RGB+Depth vs. full RGB-DF) partially mitigate this by isolating the 4D modality contribution within the same backbone, the comparison against ResNet-18 conflates representation quality with model capacity. The authors should acknowledge this confound or provide a more comparably-scaled 2D baseline.
  3. §4.2, Table 4: The world model evaluation uses only 50 held-out video sequences. This is a very small sample size for reporting metrics with three decimal places (e.g., AbsRel=0.310, AEPE=0.170) and claiming significant advantages over baselines. The variance of these metrics is unknown and could be substantial. The authors should report confidence intervals or standard deviations, and ideally expand the test set to provide a more robust evaluation.
minor comments (5)
  1. §3.5: The claim of 'high-frequency, closed-loop control' at ~9 Hz is somewhat overstated given that the system executes 10 actions open-loop over ~1.1s while the next plan is computed. The distinction between planning frequency (~0.9 Hz) and effective control frequency (~9 Hz) is noted but could be clarified to avoid misleading impressions about the system's reactivity.
  2. §3.1: The depth quantization to 8-bit grayscale (I = ⌊d/dmax × 255⌋) with a global range of [0.0, 5.0] meters may lose fine-grained depth precision, which is critical for the dexterous manipulation tasks evaluated. The authors should discuss whether this quantization limits the geometric accuracy of the policy.
  3. Table 4: The AEPE metric is computed in 'normalized RGB color space' between color-coded flow maps rather than in the native flow space. This is an unusual choice that may not accurately reflect the true optical flow error. The authors should justify this decision or provide native-space AEPE for comparison.
  4. §4.1.2: The choice of diffusion timestep t=500 for feature extraction is stated without justification. Was this value empirically optimized? A brief ablation or rationale would strengthen the paper.
  5. Figures: Several figures (e.g., Fig. 4, 6) are dense and could benefit from larger fonts or clearer labeling of components for readability.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the careful and constructive review. All three major comments identify legitimate gaps in our experimental presentation that we will address in the revision. Below we respond point by point.

read point-by-point responses

  1. Referee: §4.3, Table 5: The headline claim of 'state-of-the-art performance' against foundation models π0 and π0.5 is potentially confounded by an embodiment mismatch. The paper evaluates these models on a 54-DOF dexterous hand system (TIANJI/WUJI), but never specifies whether π0/π0.5 were fine-tuned on the 200 episodes/task used for RynnWorld-4D-Policy or run zero-shot. If zero-shot, the low scores reflect embodiment mismatch rather than a limitation of 2D representations. The paper must explicitly state the training protocol for these baselines to ensure data and compute are controlled.

    Authors: The referee is correct that the manuscript does not explicitly state the training protocol for π0 and π0.5, and this omission undermines the SOTA claim. To clarify: π0 and π0.5 were fine-tuned on the same per-task demonstration data (200 episodes/task) used for RynnWorld-4D-Policy, using the official fine-tuning recipes released by the respective authors. We adapted the action space mapping from the 54-DOF TIANJI/WUJI system to the format expected by each model. However, we acknowledge that even with fine-tuning, these models were pretrained predominantly on parallel-jaw gripper data and lack dexterous hand priors, which introduces a pretraining-data mismatch that is distinct from a pure representation-quality comparison. We agree this distinction must be made explicit. In the revision we will: (1) add a clear description of the fine-tuning protocol for all baselines in §4.2; (2) reframe the comparison as evaluating 2D-pretrained VLA models after task-specific fine-tuning on a novel embodiment, rather than as a controlled representation comparison; and (3) soften the 'state-of-the-art' language to reflect that the comparison is between our 4D-representation pipeline and 2D-pretrained VLAs adapted to this embodiment, not a controlled isolation of representation quality alone. revision: yes

  2. Referee: §4.3, Table 5: The 'w/o RynnWorld-4D' ablation uses a ResNet-18 backbone, which is drastically weaker in scale and pretraining than the proposed 5B-parameter diffusion transformer. This makes it impossible to isolate whether the 4D representation or the model scale drives the gains. The authors should acknowledge this confound or provide a more comparably-scaled 2D baseline.

    Authors: This is a fair criticism. The ResNet-18 ablation was intended to show that static 2D features are insufficient compared to predictive 4D latents, but it does conflate model scale with representation quality. We agree that this comparison alone cannot isolate the contribution of the 4D representation from the contribution of model capacity. The within-backbone ablations (RGB vs. RGB+Depth vs. RGB+Optical Flow vs. full RGB-DF, all using the same 5B RynnWorld-4D backbone) are the controlled comparisons that isolate the modality contribution, and we will emphasize these as the primary evidence for the value of 4D representations. In the revision we will: (1) explicitly acknowledge the scale confound in the ResNet-18 comparison; (2) reposition the within-backbone ablations as the main evidence for the 4D representation hypothesis; and (3) note that a comparably-scaled 2D-only baseline (e.g., the same 5B backbone trained on RGB-only video) is a natural next step, and we will add the RGB-only within-backbone result from Table 5 as the closest available controlled comparison. We will not claim that the ResNet-18 comparison isolates representation quality. revision: yes

  3. Referee: §4.2, Table 4: The world model evaluation uses only 50 held-out video sequences. This is a very small sample size for reporting metrics with three decimal places and claiming significant advantages over baselines. The authors should report confidence intervals or standard deviations, and ideally expand the test set.

    Authors: The referee is correct that 50 sequences is a small sample for the precision of reporting and the strength of the claims made. We will address this in two ways. First, we will expand the held-out test set from 50 to at least 200 sequences to provide more robust estimates. Second, we will report standard deviations (or 95% confidence intervals) for all metrics in Table 4. We will also adjust the number of reported decimal places to be consistent with the uncertainty. We acknowledge that without variance estimates, the current claims of 'significant advantages' are not statistically substantiated. The revision will include the expanded evaluation with uncertainty quantification, and we will temper any comparative language accordingly where overlaps in confidence intervals exist. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; one minor self-citation pattern in prior-work framing is not load-bearing for the central claims.

full rationale

The paper's central claims rest on two pillars: (1) the RynnWorld-4D world model, which is trained to co-generate RGB, depth, and optical flow via a tri-branch diffusion architecture (Eq. 7), and (2) the RynnWorld-4D-Policy, which consumes the frozen world model's internal latents via an inverse dynamics head (Eq. 8). Neither pillar is circular by construction. The world model is trained against pseudo-labels from external models (Depth Anything 3, DPFlow) on external datasets (Epic-Kitchens, RoboMIND, etc.), and its generation quality is evaluated against held-out test sequences (Tab. 4) using standard metrics (PSNR, SSIM, LPIPS, AbsRel, AEPE). The policy is trained on real robot demonstrations and evaluated on 35 real-world trials per task (Tab. 5). The ablation in Table 5 (RGB → RGB+Depth → RGB+Flow → full RGB-DF) uses the same frozen backbone across rows, isolating the modality contribution within the same architecture. The 'w/o RynnWorld-4D' ablation uses ResNet-18, which is a confound (model scale vs. 4D representation), but this is a correctness/experimental design concern, not circularity. The SOTA comparison against π0/π0.5 has a potential embodiment-mismatch confound (whether baselines were fine-tuned on the 54-DOF TIANJI/WUJI platform is unspecified), but again this is an experimental fairness issue, not a circular derivation. The authors cite their own prior work (Zhao et al., 2026b, 2025a, 2025b, 2024a, 2024b) in related work and introduction, but these citations provide context, not load-bearing mathematical premises. No equation, prediction, or 'first-principles result' reduces to its own inputs by construction. The 3D scene flow derivation (Eqs. 1-2) is a standard geometric unprojection, not a self-referential definition. Score 2 reflects minor self-citations that are non-load-bearing.

Axiom & Free-Parameter Ledger

8 free parameters · 4 axioms · 3 invented entities

The paper introduces several hand-tuned hyperparameters (loss weights, dropout rates, gate initialization, diffusion timestep for feature extraction, action chunk size) without systematic justification or sensitivity analysis. The most consequential axiom is the reliance on pseudo-labels from off-the-shelf models (Depth Anything 3, DPFlow) as training targets—the world model learns to imitate these models' outputs, and the 'geometric accuracy' evaluation (Tab. 4) measures against ground-truth that is itself pseudo-labeled. The pinhole camera assumption for scene flow derivation is standard but assumes known intrinsics. The claim that t=500 is the right timestep for feature extraction is ad hoc and unvalidated.

free parameters (8)
  • λ_rgb, λ_depth = 1.0
    Modality loss weights, set to 1.0 throughout all stages (Sec. 3.3).
  • λ_flow (Stage 1) = 0.5
    Flow loss weight in Stage 1, set to 0.5 because 'the flow first frame carries no informative signal at warm-up' (Sec. 3.3). Changed to 1.0 in Stages 2-3.
  • p_drop (Stage 2) = 0.2
    Branch dropout probability in Stage 2, reduced to 0.1 in Stage 3 (Sec. 3.3, Tab. 2).
  • g_m (gate initialization) = 1.0
    Learnable scalar gate in Joint Cross-Modal Attention, initialized to 1 (Eq. 6, Sec. 3.3).
  • τ (depth edge filter threshold) = not specified
    Threshold for depth-gradient-based edge filter in 3D scene reconstruction (Sec. 3.2). Value not given.
  • t=500 (diffusion timestep for feature extraction) = 500
    Diffusion timestep at which RynnWorld-4D-Policy extracts intermediate hidden states from block 15 (Sec. 4.1.2). Chosen without justification.
  • K=10 (action chunk size) = 10
    Number of future actions predicted per forward pass (Sec. 3.5).
  • N=4 (ODE steps for action generation) = 4
    Number of Euler ODE sampling steps for the flow matching policy head (Sec. 3.4).
axioms (4)
  • domain assumption Pinhole camera model: depth + optical flow can be back-projected into 3D scene flow via K^{-1} and standard unprojection (Eq. 1-2).
    Invoked in Sec. 3.2 to derive metric scene flow from RGB-DF. Standard but assumes known camera intrinsics K and no lens distortion.
  • domain assumption Pseudo-labels from Depth Anything 3 and DPFlow are sufficiently accurate to serve as training targets for a 4D world model.
    The entire Rynn4DDataset 1.0 (254M frames) uses these off-the-shelf models for depth and flow annotations (Sec. 3.1). No validation of pseudo-label accuracy on the specific dataset is provided.
  • ad hoc to paper A single forward pass through the frozen world model at diffusion timestep t=500 yields latent features that encode sufficient predictive 4D dynamics for policy learning.
    Sec. 4.1.2 states features are extracted at t=500 from block 15. No justification for this specific timestep or layer is given, and no sensitivity analysis is provided.
  • ad hoc to paper Executing 10 actions open-loop over ~1.1s while the next plan is computed in parallel is sufficient for 'high-frequency, closed-loop control.'
    Sec. 3.5 claims 9 Hz effective control frequency. The robot executes cached actions at 50 Hz but the plan only refreshes at ~0.9 Hz. Whether this constitutes 'closed-loop' depends on task dynamics.
invented entities (3)
  • Rynn4DDataset 1.0 independent evidence
    purpose: 254.4M-frame training dataset with pseudo-labeled depth, optical flow, and captions
    The dataset is described in detail (Sec. 3.1, Fig. 2-3) with sources and annotation pipeline. Links to repositories are provided. However, whether the full dataset is publicly downloadable is not confirmed in the paper.
  • Joint Cross-Modal Attention (JA) module independent evidence
    purpose: Cross-modal consistency mechanism with shared K/V, 3D RoPE, and tanh gate
    The module is defined by equations (Eqs. 3-6) and ablated in Tab. 4 ('w/o MA', 'w/o RoPE in JA'). Its contribution is empirically measured.
  • Flow Former no independent evidence
    purpose: Compresses 4D features via learnable queries with spatial cross-attention and temporal self-attention
    Defined in Eq. 8 but not ablated separately. Its contribution relative to a simpler pooling or projection is not measured.

pith-pipeline@v1.1.0-glm · 26742 in / 3960 out tokens · 416034 ms · 2026-07-08T01:45:39.813804+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages · 33 internal anchors

  1. [1]

    Cosmos World Foundation Model Platform for Physical AI

    Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    World Simulation with Video Foundation Models for Physical AI

    Arslan Ali, Junjie Bai, Maciej Bala, Yogesh Balaji, Aaron Blakeman, Tiffany Cai, Jiaxin Cao, Tianshi Cao, Eliza- beth Cha, Yu-Wei Chao, et al. World simulation with video foundation models for physical ai.arXiv preprint arXiv:2511.00062,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Qwen3-VL Technical Report

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation

    Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani. Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation.arXiv preprint arXiv:2409.16283,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Motus: A Unified Latent Action World Model

    Hongzhe Bi, Hengkai Tan, Shenghao Xie, Zeyuan Wang, Shuhe Huang, Haitian Liu, Ruowen Zhao, Yao Feng, Chendong Xiang, Yinze Rong, et al. Motus: A unified latent action world model.arXiv preprint arXiv:2512.13030,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models

    Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models.arXiv preprint arXiv:2310.10639,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al.π0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems

    Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503.06669,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    4DNeX: Feed-Forward 4D Generative Modeling Made Easy

    Zhaoxi Chen, Tianqi Liu, Long Zhuo, Jiawei Ren, Zeng Tao, He Zhu, Fangzhou Hong, Liang Pan, and Ziwei Liu. 4dnex: Feed-forward 4d generative modeling made easy.arXiv preprint arXiv:2508.13154,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Video Language Planning

    Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B Tenenbaum, et al. Video language planning.arXiv preprint arXiv:2310.10625, 2023a. Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-g...

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Articulat3D: Reconstructing Articulated Digital Twins From Monocular Videos with Geometric and Motion Constraints

    Lijun Guo, Haoyu Zhao, Xingyue Zhao, Rong Fu, Linghao Zhuang, Siteng Huang, Zhongyu Li, and Hua Zou. Articulat3d: Reconstructing articulated digital twins from monocular videos with geometric and motion constraints. arXiv preprint arXiv:2603.11606,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    World Models

    David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution.Proc. of Advances in Neural Information Processing Systems, 31, 2018a. David Ha and Jürgen Schmidhuber. World models.arXiv preprint arXiv:1803.10122, 2018b. Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent im...

    work page internal anchor Pith review Pith/arXiv arXiv 1912

  13. [13]

    TD-MPC2: Scalable, Robust World Models for Continuous Control

    Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control.arXiv preprint arXiv:2310.16828,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations

    Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. arXiv preprint arXiv:2412.14803,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al.π0.5: A vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054,

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    Galaxea Open-World Dataset and G0 Dual-System VLA Model

    Hanxiao Jiang, Hao-Yu Hsu, Kaifeng Zhang, Hsin-Ni Yu, Shenlong Wang, and Yunzhu Li. Phystwin: Physics-informed reconstruction and simulation of deformable objects from videos. InProc. of IEEE Conf. on Computer Vision and Pattern Recognition, pages 7219–7230, 2025a. Tao Jiang, Tianyuan Yuan, Yicheng Liu, Chenhao Lu, Jianning Cui, Xiao Liu, Shuiqi Cheng, Ji...

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    HunyuanVideo: A Systematic Framework For Large Video Generative Models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603,

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    Novaflow: Zero-shot manipulation via actionable flow from generated videos.arXiv preprint arXiv:2510.08568, 2025a

    Hongyu Li, Lingfeng Sun, Yafei Hu, Duy Ta, Jennifer Barry, George Konidaris, and Jiahui Fu. Novaflow: Zero-shot manipulation via actionable flow from generated videos.arXiv preprint arXiv:2510.08568, 2025a. Lin Li, Qihang Zhang, Yiming Luo, Shuai Yang, Ruilin Wang, Fei Han, Mingrui Yu, Zelin Gao, Nan Xue, Xing Zhu, et al. Causal world modeling for robot c...

    work page arXiv

  19. [19]

    Dreamitate: Real-World Visuomotor Policy Learning via Video Generation

    Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. Megasam: Accurate, fast and robust structure and motion from casual dynamic videos. InProc. of IEEE Conf. on Computer Vision and Pattern Recognition, pages 10486–10496, 2025b. Junbang Liang, Ruoshi Liu, Ege Ozguroglu, Sr...

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    Depth Anything 3: Recovering the Visual Space from Any Views

    Haotong Lin, Sili Chen, Junhao Liew, Donny Y Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang. Depth anything 3: Recovering the visual space from any views.arXiv preprint arXiv:2511.10647,

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    Flow Matching for Generative Modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling.arXiv preprint arXiv:2210.02747,

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation

    Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation.arXiv preprint arXiv:2410.07864,

    work page internal anchor Pith review Pith/arXiv arXiv

  23. [23]

    Learning Mesh-Based Simulation with Graph Networks

    Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W Battaglia. Learning mesh-based simulation with graph networks.arXiv preprint arXiv:2010.03409,

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  24. [24]

    Wan: Open and Advanced Large-Scale Video Generative Models

    Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025a. Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volu...

    work page internal anchor Pith review Pith/arXiv arXiv

  25. [25]

    Continuous 3d perception model with persistent state

    Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state. InProc. of IEEE Conf. on Computer Vision and Pattern Recognition, pages 10510–10522, 2025b. Qineng Wang, Wenlong Huang, Yu Zhou, Hang Yin, Tianwei Bao, Jianwen Lyu, Weiyu Liu, Ruohan Zhang, Jiajun Wu, Li Fei-Fei, et a...

    work page arXiv

  26. [26]

    RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation

    Kun Wu, Chengkai Hou, Jiaming Liu, Zhengping Che, Xiaozhu Ju, Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang, et al. Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation. arXiv preprint arXiv:2412.13877,

    work page internal anchor Pith review Pith/arXiv arXiv

  27. [27]

    RoboCOIN: An Open-Sourced Bimanual Robotic Data Collection for Integrated Manipulation

    Rundi Wu, Ruiqi Gao, Ben Poole, Alex Trevithick, Changxi Zheng, Jonathan T Barron, and Aleksander Holynski. Cat4d: Create anything in 4d with multi-view video diffusion models. InProc. of IEEE Conf. on Computer Vision and Pattern Recognition, pages 26057–26068, 2025a. Shihan Wu, Xuecheng Liu, Shaoxuan Xie, Pengwei Wang, Xinghang Li, Bowen Yang, Zhe Li, Ka...

    work page internal anchor Pith review Pith/arXiv arXiv

  28. [28]

    Pandora: Towards General World Model with Natural Language Actions and Video States

    Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, et al. Pandora: Towards general world model with natural language actions and video states.arXiv preprint arXiv:2406.09455,

    work page internal anchor Pith review Pith/arXiv arXiv

  29. [29]

    CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072,

    work page internal anchor Pith review Pith/arXiv arXiv

  30. [30]

    Latent Action Pretraining from Videos

    Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, et al. Latent action pretraining from videos.arXiv preprint arXiv:2410.11758,

    work page internal anchor Pith review Pith/arXiv arXiv

  31. [31]

    World Action Models are Zero-shot Policies

    Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chuning Zhu, Jiannan Xiang, et al. World action models are zero-shot policies.arXiv preprint arXiv:2602.15922,

    work page internal anchor Pith review Pith/arXiv arXiv

  32. [32]

    DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection

    Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection.arXiv preprint arXiv:2203.03605,

    work page internal anchor Pith review Pith/arXiv arXiv

  33. [33]

    Particle-grid neural dynamics for learning deformable object models from rgb-d videos.arXiv preprint arXiv:2506.15680,

    Kaifeng Zhang, Baoyu Li, Kris Hauser, and Yunzhu Li. Particle-grid neural dynamics for learning deformable object models from rgb-d videos.arXiv preprint arXiv:2506.15680,

    work page arXiv

  34. [34]

    Topology-aware Human Avatars with Semantically-guided Gaussian Splatting

    Haoyu Zhao, Chen Yang, Hao Wang, Xingyue Zhao, and Wei Shen. Sg-gs: Photo-realistic animatable human avatars with semantically-guided gaussian splatting.arXiv preprint arXiv:2408.09665, 2024a. Haoyu Zhao, Xingyue Zhao, Lingting Zhu, Weixi Zheng, and Yongchao Xu. Hfgs: 4d gaussian splatting with emphasis on spatial and temporal high-frequency components fo...

    work page internal anchor Pith review Pith/arXiv arXiv

  35. [35]

    Open-Sora: Democratizing Efficient Video Production for All

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all.arXiv preprint arXiv:2412.20404,

    work page internal anchor Pith review Pith/arXiv arXiv

  36. [36]

    Pai-bench: A comprehensive benchmark for physical ai.arXiv preprint arXiv:2512.01989,

    Fengzhe Zhou, Jiannan Huang, Jialuo Li, Deva Ramanan, and Humphrey Shi. Pai-bench: A comprehensive benchmark for physical ai.arXiv preprint arXiv:2512.01989,

    work page arXiv

  37. [37]

    IRASim: A Fine-Grained World Model for Robot Manipulation

    Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. Irasim: Learning interactive real-robot action simulators.arXiv preprint arXiv:2406.14540,

    work page internal anchor Pith review Pith/arXiv arXiv

  38. [38]

    ViT-B/16 features between the first frame and all subsequent frames. • I2V-Subject (Subj.):For image-to-video generation, this metric evaluates how faithfully the model preserves the identity of the input reference image throughout the generated sequence. It is computed as the DINO feature similarity between the conditioning image and each generated frame...

    work page 2024