Pith. sign in

REVIEW 3 major objections 6 minor 18 cited by

TesserAct: Learning 4D Embodied World Models

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TesserAct learns a 4D embodied world model as RGB-DN video generation, and the reconstructed point clouds improve robot manipulation policies over 2D video world models.

desk verdict TesserAct is a credible engineering step in 3D-aware world models, but the abstract's 'significantly outperforms' claim rests on a Table 3 that lacks error bars and cannot support it. read the letter →

arxiv 2504.20995 v1 pith:EEEHFAGT submitted 2025-04-29 cs.CV cs.RO

Haoyu Zhen , Qiao Sun , Hongxin Zhang , Junyan Li , Siyuan Zhou , Yilun Du , Chuang Gan This is my paper · ORCID
classification cs.CVcs.RO
keywords 4DworldmodelsRGB-DNvideogenerationdiffusiondepthnormalintegrationinversedynamicsrobotmanipulationnovelviewsynthesisembodiedAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TesserAct's central proposal is that a robot's world model can be a video generator that predicts depth and surface-normal maps alongside RGB, rather than an explicit 3D simulation. The paper argues that this RGB-DN video representation is rich enough to reconstruct coherent 3D scenes over time, and that the reconstructed geometry gives an inverse-dynamics policy the spatial information it needs for manipulation. To train such a model, the authors annotate existing robot manipulation videos with off-the-shelf depth and normal estimators, fine-tune a pretrained video diffusion transformer, and lift the predicted RGB-DN videos into 4D point clouds using normal integration plus two temporal-consistency losses. On RLBench, the resulting policy outperforms a 2D video world model baseline on seven of nine manipulation tasks, while the same model also produces efficient novel view synthesis.

What carries the argument

The load-bearing machinery is the RGB-DN video representation: each generated frame carries color, depth, and surface normals, which together act as a compact stand-in for a 4D scene. A latent video diffusion transformer is fine-tuned with separate input projectors for the three modalities and output branches that predict denoised depth and normal maps alongside color, preserving the pretrained RGB generation while adding geometry. The reconstruction stage then refines each depth map by integrating the predicted normals under a perspective-camera constraint, and couples frames using optical-flow-derived masks: a temporal consistency loss aligns dynamic and background regions across adjacent frames, and a regularization loss keeps the optimized depth near the generated depth. These pieces convert per-frame generated maps into a single space-time coherent point cloud that carries the argument from pixels to action.

What would settle it

On held-out RT-1 and Bridge scenes, capture the same tabletop configurations with a calibrated depth sensor and compare the reconstructed point clouds against those produced by the paper's auto-annotation pipeline; if the Chamfer distance in the grasping region is large, or if retraining the inverse-dynamics policy on measured geometry changes the RLBench success spread, the central dependence on off-the-shelf annotations is falsified.

Watch

Extended reading notes

Core claim

The paper's claim, stated in its own terms, is that a conditional RGB-DN video diffusion model is a viable 4D embodied world model. Given the current image, depth map, normal map, and a text instruction, the model generates future RGB-DN videos; because depth and normal are produced jointly with color rather than estimated from the generated video afterward, the predicted geometry is more accurate and yields lower Chamfer distances on reconstructed point clouds than RGB-only baselines. The reconstructed 4D scenes are temporally coherent, support novel view synthesis from a monocular input, and can be converted into action sequences through a PointNet-based inverse dynamics model. Across nine RLBench tasks, the policy learned from these 4D scenes outperforms the re-implemented UniPi video world model on seven of them.

Load-bearing premise

The real-world half of the training data is labeled by automatic depth and normal estimators, and the entire pipeline assumes these auto-generated labels are accurate enough to serve as ground truth for geometry and for the point clouds that drive action prediction.

Editorial extensions

If this is right

  • Video-based world models can be turned into 4D scene models by adding depth and normal channels, avoiding per-scene optimization of explicit 4D neural representations.
  • Jointly predicting depth and normal with color produces lower point-cloud reconstruction error than predicting color first and estimating geometry afterward.
  • An inverse-dynamics policy trained on reconstructed point-cloud states outperforms a 2D video world model baseline on seven of nine RLBench manipulation tasks.
  • The same RGB-DN video generation supports novel view synthesis, matching or beating a Gaussian-splatting video reconstruction method in quality while taking about one minute instead of two hours.
  • Existing 2D robot video datasets can be converted into 4D embodied training data with off-the-shelf depth and normal estimators.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Since the RGB-DN representation only captures a single visible surface, a natural extension is multi-view RGB-DN generation; the paper's own limitation note points to this, and integrating several predicted views could yield closed 4D scenes rather than front-facing shells.
  • The optical-flow-based temporal consistency loss and normal-integration refinement are not specific to robot videos, so the same recipe could be applied to general monocular video-to-4D reconstruction tasks where depth, normal, and flow estimators already exist.
  • A testable stress test is whether the framework's advantage persists when point clouds are built from ground-truth metric depth instead of auto-estimated labels; this would separate the contribution of the RGB-DN world model architecture from the contribution of the annotation pipeline.
  • If auto-annotated geometry is systematically biased, retraining with a modest amount of real metric-depth video could restore the gains; the architecture itself does not depend on the particular estimator.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. TesserAct proposes to learn a 4D embodied world model by training a video diffusion model to jointly generate RGB, depth, and normal videos from a current frame and a language instruction. The paper collects a dataset of simulated RLBench videos with ground-truth depth and synthesized normals, plus real-world Fractal, Bridge, and Something-SomethingV2 videos annotated with off-the-shelf depth and normal estimators. It fine-tunes CogVideoX with additional depth/normal input/output projectors, reconstructs 4D point clouds by normal-integrated depth optimization with optical-flow-based consistency and regularization losses, and uses a PointNet-based inverse dynamics model on the point clouds for manipulation. Experiments report improved depth/normal metrics, Chamfer distances, novel view synthesis, and success rates on 9 RLBench tasks compared to UniPi* and Image-BC baselines.

Significance. The approach is well-motivated and the RGB-DN video representation is a pragmatic intermediate for 3D-aware prediction. The measured improvements on synthetic RLBench depth/normal quality and Chamfer distance (e.g., 0.0811 vs 0.2570 for OpenSora in Table 2) are substantial, and the qualitative generalization to unseen scenes and embodiments is promising. The main weakness is that the headline policy-learning claim is not supported with statistical significance: Table 3 reports 100-episode success rates without error bars, confidence intervals, or multiple seeds, and the 7-of-9 advantage over UniPi* is within sampling noise for most tasks. The quantitative ablation of the proposed losses is also missing. With these issues addressed, the paper would be a solid contribution.

major comments (3)
  1. [5.2, Table 3] The abstract claims TesserAct 'significantly outperforms' prior video-based world models, and Table 3 is the sole empirical support. Success rates are averaged over 100 episodes without standard deviations, confidence intervals, or repeated seeds. For binary outcomes, the standard error of a success rate is at most about 5 percentage points, and the standard error of the difference between two such rates is roughly 6–7 points. Thus the reported gaps are mostly not statistically significant (e.g., close box 88 vs 81, z≈1.4; put knife 70 vs 66; water plants 41 vs 35), and even a one-sided sign test on 7 wins out of 9 gives p≈0.09. Please report per-seed means, confidence intervals, or significance tests, and soften the claim accordingly.
  2. [5.1.2, Figure 3 / Sec. 4.3] The consistency and regularization losses are introduced as novel contributions, but the ablation is limited to qualitative images in Figure 3. No quantitative comparison (e.g., Chamfer distance, depth AbsRel, or normal error with and without each loss) is provided. Without numbers, the claim that 'Consistency and Regularization Loss are effective' cannot be assessed. Please add a quantitative ablation table, at least on the RLBench subset where ground truth is available.
  3. [4.1, Table 2] Real-domain depth and normal annotations are produced by RollingDepth and Marigold-LCM, and Table 2 evaluates the real-domain results against these same off-the-shelf estimates. This setup cannot detect systematic bias in those estimators; if they are inaccurate on robot manipulation scenes, the reported 'high-quality 4D scenes' may reflect modeling of estimator artifacts rather than true geometry. Please validate on a subset with sensor ground truth (e.g., a depth camera) or an independent estimator, and state the limitation in the paper. This does not invalidate the RLBench policy comparison, which uses simulator ground truth, but it qualifies the real-domain 4D claims.
minor comments (6)
  1. [3.2, Eq. (3)] The second quadratic term in Eq. (3) repeats \(\partial_u\tilde{d}\); it should presumably be \(\partial_v\tilde{d}\).
  2. [5.1.1] The dataset name 'Bridage' should be 'Bridge'.
  3. [Table 2] The column labeled '11.25◦' should specify that it is the percentage of pixels within 11.25° (higher is better), and the SSIM values appear to be percentages; please clarify the units.
  4. [4.3, Eq. (5)] The text defines the static mask with 'smaller than threshold c' while Eq. (5) uses \(\le c\); please make the inequality consistent.
  5. [Supplementary Table 5] The supplementary text refers to 'Eq.12' when describing the loss parameters, but the main-text loss objective is Eq. (7). Please correct the cross-reference.
  6. [General] The paper does not state whether code and trained models will be released; for reproducibility, please include a release statement or explicitly note why this is not possible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation chain is self-contained and every evaluated quantity is measured against external or independently annotated targets.

full rationale

The paper's central derivation is a standard supervised pipeline: (i) it builds a 4D video dataset with synthetic RLBench ground truth plus off-the-shelf depth/normal annotations, (ii) it fine-tunes CogVideoX to denoise RGB-DN latents with the usual diffusion objective (Eq. 4), and (iii) it converts generated RGB-DN videos into point clouds by optimizing the normal-integration loss (Eq. 3) with cross-frame consistency and regularization terms (Eqs. 5-7). None of these equations reduces by construction to the reported outcome: the video diffusion loss is a standard denoising objective against dataset targets, the reconstruction losses are optimization objectives over generated geometry, and the downstream policy success (Table 3) is measured in the RLBench simulator with ground-truth dynamics, not derived from any fitted parameter of the model. The pseudo-label concern for real-domain data (RollingDepth and Marigold-LCM annotations used both for training and evaluation) is a data-quality and external-validity issue, not a circularity: the model is not fitting a parameter and then reporting that same fit as a prediction. The statistical weakness of Table 3 (no error bars or significance tests) is a correctness risk, not a circular structure. Self-citations to prior work by the same authors (e.g., UniPi, 3D-VLA) appear as background or as baselines, and no load-bearing claim is justified only by a self-citation; no uniqueness theorem is invoked, and no ansatz is smuggled in through citation. The paper is therefore self-contained against external benchmarks for the purposes of this pass.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The main free parameters are the tuned reconstruction loss weights and an unreported flow threshold. The load-bearing axioms are the reliability of off-the-shelf depth and normal annotations, the transferability of the pretrained video prior to new channels, and the correctness of the optical-flow-based segmentation. No new physical entities are introduced.

free parameters (2)
  • Reconstruction loss weights (lambda_d, lambda_b, lambda_g1, lambda_g2) = RT-1/Bridge: 20, 200, 20, 20; RLBench: 20, 200, 2, 2
    Reported in Supplementary Table 5 and described as needing per-dataset tuning; these weights directly scale the spatial, consistency, and regularization terms in Eq. 7 and hence control the geometry of the reconstructed 4D scenes that feed the downstream policy.
  • Optical flow static/dynamic threshold c = not reported
    Defined in Sec 4.3 as the cutoff for static regions in the flow mask (M_s = ||F_i|| <= c). The value is never given, yet it determines which pixels receive background versus dynamic consistency constraints.
assumptions (4)
  • ad hoc to paper Off-the-shelf depth and normal estimators (RollingDepth, Marigold-LCM, DSINE) produce annotations accurate enough to serve as training targets.
    Introduced in Sec 4.1; no validation of estimator accuracy on robotic data is provided, so the quality of the training signal is unverified.
  • standard math The normal-integration optimization (Eq. 3) plus consistency and regularization losses recovers a geometrically consistent 4D scene from relative depth maps.
    Adopted from prior normal integration literature; the paper does not address global scale ambiguity of relative depth, which limits metric accuracy of reconstructed scenes.
  • domain assumption RAFT optical flow reliably separates static and dynamic regions in generated videos.
    Used to define masks in Eq. 5; occlusions and large motions can break flow correspondence, and no failure cases are analyzed.
  • domain assumption CogVideoX's pretrained video prior transfers to the new depth and normal branches with zero-initialized adapters.
    Architecture in Sec 4.2 relies on this transfer; no ablation compares against training the depth and normal branches from scratch.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TesserAct: Learning 4D Embodied World Models." pith.science (2026). https://pith.science/paper/EEEHFAGT

@misc{pith2026250420995,
  author       = {Pith},
  title        = {Pith review of: TesserAct: Learning 4D Embodied World Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EEEHFAGT}},
  note         = {Machine review of arXiv:2504.20995}
}
read the original abstract

This paper presents an effective approach for learning novel 4D embodied world models, which predict the dynamic evolution of 3D scenes over time in response to an embodied agent's actions, providing both spatial and temporal consistency. We propose to learn a 4D world model by training on RGB-DN (RGB, Depth, and Normal) videos. This not only surpasses traditional 2D models by incorporating detailed shape, configuration, and temporal changes into their predictions, but also allows us to effectively learn accurate inverse dynamic models for an embodied agent. Specifically, we first extend existing robotic manipulation video datasets with depth and normal information leveraging off-the-shelf models. Next, we fine-tune a video generation model on this annotated dataset, which jointly predicts RGB-DN (RGB, Depth, and Normal) for each frame. We then present an algorithm to directly convert generated RGB, Depth, and Normal videos into a high-quality 4D scene of the world. Our method ensures temporal and spatial coherence in 4D scene predictions from embodied scenarios, enables novel view synthesis for embodied environments, and facilitates policy learning that significantly outperforms those derived from prior video-based world models.

Figures

Figures reproduced from arXiv: 2504.20995 by the authors.

Figure 1
Figure 1. We propose TesserAct, the 4D Embodied World Model, which takes an input image and text instruction to generate RGB, depth, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Architecture and Training Overview of TesserAct. We propose to learn a 4D embodied world model, TesserAct, by training on RGB-DN videos and reconstructing the 4D scenes from it. We first introduce the 4D embodied video dataset we collected in Sec. 4.1, then discuss the model architecture and training strategy in Sec. 4.2. In Sec. 4.3, we propose an efficient optimization algorithm with two novel loss functions to co… view at source ↗
Figure 3
Figure 3. Effect of consistency and regularization loss on 4D scene reconstruction. The red boxes highlight the inconsistent regions. representations are perturbed with noise to get xt, and are then fed into our model along with the corresponding image latent head x 0 . For the input design, we introduce three separate projectors for each modality to extract the embed￾dings: fz = InputProj(zt, z 0 ), where z ∈ {v, d, n}. DiT … view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Qualitative results of (a) In-domain 4D generation results. (b) Generalization over unseen scenes and objects. (c) Novel view synthesis [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Visualization of the optimized 3D robotic scene recon [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Comparison of point cloud generation quality between [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 8
Figure 8. Figure 8: Tracking and visualization of robotic arm action trajecto [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: In-domain RGB-DN Video generation results on Bridge dataset [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: In-domain RGB-DN Video generation results on RT1 dataset [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: In-domain RGB-DN Video generation results on RLBench dataset [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Some sample frames extracted from the datasets Bridge [ [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Masked Visual Actions for Unified World Modeling

    cs.CV 2026-07 conditional novelty 7.0 of 10

    A single video model finetuned on masked pixel trajectories acts as both forward and inverse robot world model, enabling policy evaluation, planning, and action extraction.

  2. HyperShadow: A Benchmark for Detecting 3D Projections of Higher-Dimensional Spatial Objects

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Shadows of 4D–6D objects projected to 3D are detectable by learned point-cloud models and by a zero-parameter rigidity residual, but not by intrinsic-dimension estimation.

  3. 4D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Time

    cs.CV 2025-06 conditional novelty 7.0 of 10

    4D-LRM is a transformer that maps sparse posed frames scattered across time to a cloud of 4D Gaussians and renders any query view at any query time in under 1.5 seconds.

  4. LAWM-3D: Learning 3D-Aware Latent Actions from Human Videos for Generalizable Robot World Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    LAWM-3D learns 3D-aware latent actions from multi-view human videos by combining VGGT geometric alignment with RGB-depth reconstruction, improving robot world model prediction and generalization.

  5. DerainSplat: Feed-Forward Clean 3D Gaussian Splatting from Sparse Rainy Views

    cs.CV 2026-08 conditional novelty 6.0 of 10

    DerainSplat reconstructs clean 3D Gaussian scenes from sparse rainy views in a single forward pass by predicting weather factors and using support maps to guide matching and appearance fusion.

  6. SG-WAM: Self-Guided World Modeling in Geometry-Aware Policy Space

    cs.RO 2026-08 conditional novelty 6.0 of 10

    A 0.9B policy that couples action generation with latent future-state prediction in its own representation space, plus geometric supervision, reaches 98.5% LIBERO and 73% LIBERO-Plus.

  7. ContactFlow: A video action conditioning that transfers across embodiments

    cs.RO 2026-07 conditional novelty 6.0 of 10

    A contact-point-based action representation lets a video world model transfer manipulation knowledge across human and robot embodiments.

  8. Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectories

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Pre-training a VLA model on 100k hours of auto-labeled UMI trajectories, then post-training on robot data, yields SOTA simulated manipulation and data-efficient fine-tuning.

  9. MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation

    cs.CV 2026-02 conditional novelty 6.0 of 10

    A geometry-consistent multi-view RGBD 4D world model for robot manipulation, whose actions are recovered by test-time optimization of a learned trajectory latent, outperforming single- and dual-view world-model baseli...

  10. 4DNeX: Feed-Forward 4D Generative Modeling Made Easy

    cs.CV 2025-08 conditional novelty 6.0 of 10

    4DNeX generates dynamic 3D point clouds and matching RGB video from a single image by fine-tuning a pretrained video diffusion model on a large pseudo-annotated 4D dataset.

  11. Quo Vadis, World Modeling?

    cs.CV 2026-08 conditional novelty 5.0 of 10

    An agent-centric reframing of world modeling, replacing physical state prediction with 'information transitions' organized into six proxy functions and three empowerment levels.

  12. BWM: A Low-Cost High-Fidelity World Simulator for Robot Learning

    cs.RO 2026-07 conditional novelty 5.0 of 10

    BWM is an action-conditioned video world model that, on the WorldArena benchmark and a six-task physical-robot test, outperforms compared simulators both as a data generator and as a policy evaluator.

  13. MoWorld: A Flash World Model

    cs.CV 2026-07 conditional novelty 5.0 of 10

    MoWorld achieves up to 50 FPS real-time interactive world simulation on NPUs by combining a 3D-native data engine, curriculum cross-frame pretraining, autoregressive distillation, and mixed-precision parallel inference.

  14. RoDyn: Taming Interactive Robot-Dynamic 2.5D World Model for Robotic Manipulation

    cs.RO 2025-10 unverdicted novelty 5.0 of 10

    Abstract describes RoDyn but full text describes iMoWM; the record is internally inconsistent and the headline claims are absent from the body.

  15. RoboScape: Physics-informed Embodied World Model

    cs.CV 2025-06 conditional novelty 5.0 of 10

    RoboScape jointly learns RGB video, depth, and keypoint-token consistency in one autoregressive world model, improving video quality, geometry, action control, synthetic-data policy training, and policy evaluation for...

  16. 3DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Model

    cs.RO 2025-06 conditional novelty 5.0 of 10

    A diffusion world model predicts 3D optical flow as an embodiment-agnostic action plan, and constrained optimization converts the flow into robot arm actions.

  17. VRAG: Learning World Models for Interactive Video Generation

    cs.CV 2025-05 unverdicted novelty 5.0 of 10

    VRAG improves long-horizon interactive video generation by conditioning autoregressive diffusion on retrieved historical frames and explicit global state, outperforming long-context baselines on the tested Minecraft a...

  18. Reinforcement Learning: From Algorithms To Foundation Models

    cs.AI 2026-07 conditional novelty 3.0 of 10

    A dissertation uniting the author's published results: non-exploitable Nash-DQN policies and the FightLadder benchmark for games, plus diffusion/consistency-model world models for RL — a compilation rather than new results.

Reference graph

Works this paper leans on

85 extracted references · 32 canonical work pages · cited by 18 Pith papers

  1. [1]

    A separation princi- ple for control in the age of deep learning

    Alessandro Achille and Stefano Soatto. A separation princi- ple for control in the age of deep learning. Annual Review of Control, Robotics, and Autonomous Systems, 1:287–307,

  2. [2]

    Gwangbin Bae and Andrew J. Davison. Rethinking inductive biases for surface normal estimation. In IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR),

  3. [3]

    4d-fy: Text-to-4d generation using hybrid score distillation sampling

    Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7996–8006,

  4. [4]

    Bertsekas

    Dimitri P. Bertsekas. Dynamic Programming and Optimal Control. Athena Scientific, 1995. 2

    1995

  5. [5]

    Zoedepth: Zero-shot trans- fer by combining relative and metric depth

    Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias M ¨uller. Zoedepth: Zero-shot trans- fer by combining relative and metric depth. arXiv preprint arXiv:2302.12288, 2023. 14

    arXiv 2023

  6. [6]

    Rt-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakr- ishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022. 3, 4, 6, 13, 18

    arXiv 2022

  7. [7]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. 2

    arXiv 2023

  8. [8]

    Genie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker- Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first Interna- tional Conference on Machine Learning, 2024. 2

    2024

Show all 85 references
  1. [9]

    Bilateral normal integration

    Xu Cao, Hiroaki Santo, Boxin Shi, Fumio Okura, and Ya- suyuki Matsushita. Bilateral normal integration. In European Conference on Computer Vision, pages 552–567. Springer,

  2. [10]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 144...

  3. [11]

    Trans- dreamer: Reinforcement learning with transformer world models

    Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Trans- dreamer: Reinforcement learning with transformer world models. arXiv preprint arXiv:2202.09481, 2022. 2

    2022 arXiv

  4. [12]

    Recurrent environment simulators

    Silvia Chiappa, S ´ebastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. arXiv preprint arXiv:1704.02254, 2017. 2

    2017 arXiv

  5. [13]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. 2

    2023 arXiv

  6. [14]

    Video language planning

    Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B Tenenbaum, et al. Video language planning. arXiv preprint arXiv:2310.10625, 2023. 2

    2023 arXiv

  7. [15]

    Learn- ing universal policies via text-guided video generation

    Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learn- ing universal policies via text-guided video generation. Ad- vances in Neural Information Processing Systems, 36, 2024. 2, 7

    2024

  8. [16]

    Integrating the normal field of a surface in the presence of discontinuities

    Jean-Denis Durou, Jean-Fran c ¸ois Aujol, and Fr ´ed´eric Courteille. Integrating the normal field of a surface in the presence of discontinuities. In International Workshop on En- ergy Minimization Methods in Computer Vision and Pattern Recognition, pages 261–273. Springer, 2009. 3

    2009

  9. [17]

    Metrics for finite markov decision processes

    Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In UAI, pages 162–169,

  10. [18]

    Foundation models in robotics: Applications, challenges, and the fu- ture

    Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, et al. Foundation models in robotics: Applications, challenges, and the fu- ture. The International Journal of Robotics Research, page ...

    2023

  11. [19]

    The” something something” video database for learning and evaluating visual common sense

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller- Freitag, et al. The” something something” video database for learning and evaluating visual common sense. In...

    2017

  12. [20]

    Generating exe- cutable action plans with environmentally-aware language models

    Maitrey Gramopadhye and Daniel Szafir. Generating exe- cutable action plans with environmentally-aware language models. arXiv preprint arXiv:2210.04964, 2022. 2

    2022 arXiv

  13. [21]

    Recurrent world models facilitate policy evolution

    David Ha and J¨urgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2018. 2

    2018

  14. [22]

    Mastering atari with discrete world models

    Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021. 2

    2021

  15. [23]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 3

    2020

  16. [24]

    3d-llm: Injecting the 3d world into large language models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482–20494, 2023. 2

    2023

  17. [25]

    An embodied generalist agent in 3d world

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baox- iong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871, 2023. 2

    2023 arXiv

  18. [26]

    Stephen James, Zicong Ma, David Rovick Arrojo, and An- drew J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters,

  19. [27]

    Bc-z: Zero-shot task generalization with robotic imitation learning

    Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Fred- erik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991–1002. PMLR, 2022. 7

    2022

  20. [28]

    Planning with diffusion for flexible behavior synthe- sis

    Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthe- sis. arXiv preprint arXiv:2205.09991, 2022. 2

    2022 arXiv

  21. [29]

    Vima: General robot manipulation with multimodal prompts

    Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anand- kumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03094, 2(3):6, 2022. 2

    arXiv

  22. [30]

    Consistent4d: Consistent 360 {\deg} dynamic object genera- tion from monocular video

    Yanqin Jiang, Li Zhang, Jin Gao, Weimin Hu, and Yao Yao. Consistent4d: Consistent 360 {\deg} dynamic object genera- tion from monocular video. arXiv preprint arXiv:2311.02848,

    arXiv

  23. [31]

    Video depth without video models, 2024

    Bingxin Ke, Dominik Narnhofer, Shengyu Huang, Lei Ke, Torben Peters, Katerina Fragkiadaki, Anton Obukhov, and Konrad Schindler. Video depth without video models, 2024. 3, 4

    2024

  24. [32]

    Repurpos- ing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos- ing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9492–9502,

  25. [33]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139–1, 2023. 3, 7

    2023

  26. [34]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024. 2

    2024 arXiv

  27. [35]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 3, 4

    2013 arXiv

  28. [36]

    Exponential moving average versus moving exponential average

    Frank Klinker. Exponential moving average versus moving exponential average. Mathematische Semesterberichte, 58: 97–107, 2011. 13

    2011

  29. [37]

    Aesthetic predictor

    LAION-AI. Aesthetic predictor. https://github.com/ LAION-AI/aesthetic-predictor, 2022. 7

    2022

  30. [38]

    State representation learning for control: An overview

    Timoth´ee Lesort, Natalia D ´ıaz-Rodr´ıguez, Jean-Franois Goudou, and David Filliat. State representation learning for control: An overview. Neural Networks, 108:379–392,

  31. [39]

    Pre-trained language models for interactive decision-making

    Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Aky ¨urek, Anima Anandkumar, et al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems, 35:31199–31212, 2022. 2

    2022

  32. [40]

    Dreamitate: Real-world visuomotor policy learning via video generation

    Junbang Liang, Ruoshi Liu, Ege Ozguroglu, Sruthi Sudhakar, Achal Dave, Pavel Tokmakov, Shuran Song, and Carl V on- drick. Dreamitate: Real-world visuomotor policy learning via video generation. arXiv preprint arXiv:2406.16862, 2024. 2

    2024 arXiv

  33. [41]

    Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models

    Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fi- dler, and Karsten Kreis. Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8576–8588, 2024. 3

    2024

  34. [42]

    Prentice-Hall, Inc., 1994

    Lennart Ljung and Torkel Glad.Modeling of dynamic systems. Prentice-Hall, Inc., 1994. 2

    1994

  35. [43]

    Trans- formers are sample efficient world models

    Vincent Micheli, Eloi Alonso, and Franc ¸ois Fleuret. Trans- formers are sample efficient world models. arXiv preprint arXiv:2209.00588, 2022. 2

    2022 arXiv

  36. [44]

    Nerf: Representing scenes as neural radiance fields for view syn- thesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. Communications of the ACM , 65(1):99–106, 2021. 3

    2021

  37. [45]

    Point-e: A system for generat- ing 3d point clouds from complex prompts

    Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generat- ing 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751, 2022. 6

    2022 arXiv

  38. [46]

    Video generation models as world simu- lators

    OpenAI. Video generation models as world simu- lators. https : / / openai . com / index / video - generation - models - as - world - simulators/,

  39. [47]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  40. [48]

    The colosseum: A bench- mark for evaluating generalization for robotic manipulation

    Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, and Dieter Fox. The colosseum: A bench- mark for evaluating generalization for robotic manipulation

  41. [49]

    Pointnet++: Deep hierarchical feature learning on point sets in a metric space

    Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017. 6

    2017

  42. [50]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  43. [51]

    Planning with large language models via corrective re-prompting

    Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. Planning with large language models via corrective re-prompting. arXiv preprint arXiv:2211.09935, 2022. 2

    2022 arXiv

  44. [52]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International confer- ence on machine learning , pages 8821–8831. Pmlr, 2021. 14

    2021

  45. [53]

    Dreamgaussian4d: Generative 4d gaussian splatting

    Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Dreamgaussian4d: Generative 4d gaussian splatting. arXiv preprint arXiv:2312.17142, 2023. 3

    2023 arXiv

  46. [54]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 3

    2022

  47. [55]

    Text-to-4d dy- namic scene generation

    Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Iurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Text-to-4d dy- namic scene generation. arXiv preprint arXiv:2301.11280,

    arXiv

  48. [56]

    Dyna, an integrated architecture for learn- ing, planning, and reacting

    Richard S Sutton. Dyna, an integrated architecture for learn- ing, planning, and reacting. ACM Sigart Bulletin, 2(4):160– 163, 1991. 2

    1991

  49. [57]

    Aether: Geometric-aware unified world modeling

    Aether Team, Haoyi Zhu, Yifan Wang, Jianjun Zhou, Wen- zheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Chunhua Shen, Jiangmiao Pang, et al. Aether: Geometric-aware unified world modeling. arXiv preprint arXiv:2503.18945, 2025. 2

    2025 arXiv

  50. [58]

    Open X-Embodiment: Robotic learning datasets and RT-X models

    Open X-Embodiment Team. Open X-Embodiment: Robotic learning datasets and RT-X models. https://arxiv. org/abs/2310.08864, 2023. 4

    2023 arXiv

  51. [59]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020. 5

    2020

  52. [60]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 3, 4

    2017

  53. [61]

    Bridgedata v2: A dataset for robot learning at scale

    Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen- Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning ...

    2023

  54. [62]

    Shape of motion: 4d reconstruc- tion from a single video

    Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruc- tion from a single video. arXiv preprint arXiv:2407.13764,

  55. [63]

    Describe, explain, plan and select: interactive planning with llms enables open-world multi-task agents

    Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: interactive planning with llms enables open-world multi-task agents. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 2

    2023

  56. [64]

    Pandora: Towards general world model with natural language actions and video states

    Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, et al. Pandora: Towards general world model with natural language actions and video states. arXiv preprint arXiv:2406.09455, 2024. 2

    2024 arXiv

  57. [65]

    Sv4d: Dynamic 3d content generation with multi-frame and multi-view consistency

    Yiming Xie, Chun-Han Yao, Vikram V oleti, Huaizu Jiang, and Varun Jampani. Sv4d: Dynamic 3d content generation with multi-frame and multi-view consistency. arXiv preprint arXiv:2407.17470, 2024. 3

    2024 arXiv

  58. [66]

    Yuliang Xiu, Jinlong Yang, Xu Cao, Dimitrios Tzionas, and Michael J. Black. ECON: Explicit Clothed humans Optimized via Normal integration. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR),

  59. [67]

    Learn- ing interactive real-world simulators

    Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learn- ing interactive real-world simulators. arXiv preprint arXiv:2310.06114, 2023. 2

    2023 arXiv

  60. [68]

    Foundation models for deci- sion making: Problems, methods, and opportunities

    Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foundation models for deci- sion making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04129, 2023. 2

    2023 arXiv

  61. [69]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 4, 6, 7, 13

    2024 arXiv

  62. [70]

    Stablenormal: Reducing diffusion variance for stable and sharp normal

    Chongjie Ye, Lingteng Qiu, Xiaodong Gu, Qi Zuo, Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu, and Xiaoguang Han. Stablenormal: Reducing diffusion variance for stable and sharp normal. arXiv preprint arXiv:2406.16864, 2024. 5

    2024 arXiv

  63. [71]

    4dgen: Grounded 4d content gener- ation with spatial-temporal consistency

    Yuyang Yin, Dejia Xu, Zhangyang Wang, Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content gener- ation with spatial-temporal consistency. arXiv preprint arXiv:2312.17225, 2023. 3

    2023 arXiv

  64. [72]

    Freeman, and Jiajun Wu

    Hong-Xing Yu, Haoyi Duan, Charles Herrmann, William T. Freeman, and Jiajun Wu. Wonderworld: Interactive 3d scene generation from a single image. arXiv:2406.09394, 2024. 5

    2024 arXiv

  65. [73]

    Building cooperative embodied agents modularly with large language models

    Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. arXiv preprint arXiv:2307.02485 ,

    arXiv

  66. [74]

    Combo: Compositional world models for embodied multi- agent cooperation

    Hongxin Zhang, Zeyuan Wang, Qiushi Lyu, Zheyuan Zhang, Sunli Chen, Tianmin Shu, Yilun Du, and Chuang Gan. Combo: Compositional world models for embodied multi- agent cooperation. arXiv preprint arXiv:2404.10775, 2024. 2

    2024 arXiv

  67. [75]

    Animate124: Animating one image to 4d dynamic scene

    Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong, Zhen- guo Li, and Gim Hee Lee. Animate124: Animating one image to 4d dynamic scene. arXiv preprint arXiv:2311.14603, 2023. 3

    2023 arXiv

  68. [76]

    3d-vla: A 3d vision-language-action generative world model

    Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d vision-language-action generative world model. arXiv preprint arXiv:2403.09631, 2024. 2, 5, 14

    2024 arXiv

  69. [77]

    Open-sora: Democratizing efficient video production for all, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024. 2, 3, 4, 6

    2024

  70. [78]

    clip-score: CLIP Score for Py- Torch

    SUN Zhengwentai. clip-score: CLIP Score for Py- Torch. https : / / github . com / taited / clip - score, 2023. Version 0.1.1. 7

    2023

  71. [79]

    Robodreamer: Learning composi- tional world models for robot imagination

    Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. Robodreamer: Learning composi- tional world models for robot imagination. arXiv preprint arXiv:2404.12377, 2024. 2

    2024 arXiv

  72. [80]

    Irasim: Learning interactive real- robot action simulators

    Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. Irasim: Learning interactive real- robot action simulators. arXiv preprint arXiv:2406.14540,

    arXiv

  73. [81]

    ˚Astr¨om and Bj¨orn Wittenmark

    Karl J. ˚Astr¨om and Bj¨orn Wittenmark. Adaptive control of linear time-invariant systems. Automatica, 9(6):551–564,

  74. [84]

    Video Diffusion Model Details We trained an RGB-DN video diffusion model using the CogVideoX [69] architecture

    Implementation Details 1.1. Video Diffusion Model Details We trained an RGB-DN video diffusion model using the CogVideoX [69] architecture. On the input side, our depth normal projector and RGB projector shared the same archi- tecture. On the output side, our Conv3DNet consist...

  75. [85]

    Data Annotation In this section, we first compare our data generation method with 3D-VLA [76]

    More Qualitative Results 2.1. Data Annotation In this section, we first compare our data generation method with 3D-VLA [76]. They use ZoeDepth [5] for depth map estimation and directly map them into 3D space. The com- parison results, shown in Figure 6, evaluate the quality of...

  76. [1973]

    2 TesserAct: Learning 4D Embodied World Models Supplementary Material

  77. [2023]

    Accessed: 2024-10-01. 2

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.