Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

Whole-Body Conditioned Egocentric Video Prediction

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PEVA predicts the next egocentric frame from past video and a 48-dimensional whole-body pose delta, and the paper reports it outperforms prior action-conditioned baselines on LPIPS, DreamSim, and FID.

desk verdict A clearly written, useful extension of egocentric world models, but the headline claim is not yet supported because the action vector leaks the future camera pose and no ablation controls for it. read the letter →

arxiv 2506.21552 v1 pith:MPJM4L73 submitted 2025-06-26 cs.CV cs.AIcs.LGcs.MMcs.RO

Yutong Bai , Danny Tran , Amir Bar , Yann LeCun , Trevor Darrell , Jitendra Malik This is my paper · ORCID
classification cs.CVcs.AIcs.LGcs.MMcs.RO
keywords egocentricvideopredictionwhole-bodyposeconditioningdiffusiontransformeraction-conditionedgenerationworldmodelsNymeriadatasetautoregressivecounterfactualplanning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that first-person video prediction is best driven by the agent's own body: given past frames and a sequence of 3D pose deltas, a model can simulate what a person would next see. The authors build PEVA, an autoregressive diffusion transformer trained on real-world egocentric video synchronized with motion capture, and report that whole-body conditioning improves perceptual and semantic prediction quality over prior action-conditioned baselines. The value of the claim, if right, is that embodied agents could preview the visible consequences of their movements, follow specified atomic actions, and choose between action sequences by simulating outcomes.

What carries the argument

The load-bearing object is the structured action representation: a 48-dimensional vector per time step built from the delta of root translation and delta rotations of 15 upper-body joints in the Xsens kinematic ordering, normalized to a pelvis-centered local frame. This action vector enters each denoising block through adaptive layer norm (AdaLN), alongside clean context tokens from past frames. The architecture is a conditional diffusion transformer trained with random timeskips, sequence-level prefix losses, and causal and spatial attention masks, so the model learns both fine-grained joint control and long-horizon dynamics.

What would settle it

Run a head/root-only ablation: condition PEVA on just the root translation and head rotation components of the action and compare LPIPS and DreamSim against full whole-body conditioning; if head/root-only conditioning captures most of the improvement over CDiT, the claim that whole-body pose dynamics drive the gain is not supported.

Watch

Extended reading notes

Core claim

PEVA conditions an autoregressive diffusion transformer on an action vector that encodes, for each step, the change in root translation and the relative rotation of every upper-body joint, organized by the kinematic tree. Rolling out the model frame by frame on Nymeria data yields lower LPIPS (0.303 vs 0.313) and DreamSim (0.193 vs 0.202) than CDiT at a 2-second horizon, with better FID, and the advantage persists over 16-second rollouts. The same conditioning also lets the model generate videos of atomic hand and whole-body motions and supports a planning loop that scores simulated action candidates by their LPIPS match to a goal image.

Load-bearing premise

The measured improvements could come largely from the action vector specifying the future camera path through root and head rotations, rather than from the model learning how the rest of the body shapes what is seen; if so, the whole-body claim would be overstated.

Editorial extensions

If this is right

  • An embodied agent can convert a planned pose trajectory into a preview of its own future camera view, supporting reach-and-grasp and navigation decisions before acting.
  • Because the action vector separates joints in the kinematic tree, the model can follow atomic motion commands such as left hand up, rotate right, or move forward without retraining.
  • Rollouts stay semantically plausible for at least 16 seconds, so multi-second lookahead for planning is feasible with this approach.
  • The planning protocol demonstrates a template: simulate several action candidates, score each generated frame against a goal image, and pick the candidate with the best match.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not ablate the head and root components of the action vector; if those alone reproduce most of the reported gain, the improvement may be due to specifying camera motion rather than whole-body dynamics.
  • Because conditioning covers only the upper body above the pelvis, extending the action space to leg and foot trajectories is a direct next step for locomotion-heavy scenes.
  • The same structured conditioning could be applied to object-centric or hand-specific world models, where predicting the visual result of a hand motion is the bottleneck.
  • If the gains hold under the head/root ablation, PEVA-style conditioning could serve as a cheap way to make existing video world models physically controllable without changing their generative backbone.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces PEVA, an autoregressive conditional diffusion transformer for egocentric video prediction conditioned on whole-body 3D pose trajectories. The action representation is a 48-dimensional vector consisting of root translation deltas and relative Euler-angle rotations of 15 upper-body joints, including the head. The model is trained on the Nymeria dataset and evaluated on single-step prediction at 2-second intervals, atomic-action control, long-horizon rollouts up to 16 seconds, and a preliminary CEM-based planning experiment. The central quantitative claim is that PEVA outperforms CDiT and a Diffusion-Forcing variant on LPIPS, DreamSim, and FID, with the abstract asserting that whole-body pose conditioning lets the model learn how physical actions shape the first-person view.

Significance. If the central claim holds, PEVA would be a meaningful step toward action-conditioned world models for embodied agents, going beyond the low-dimensional navigation controls used in prior work such as Navigation World Models. The paper contributes a structured kinematic action representation, sequence-level training with random timeskips, and a hierarchical evaluation protocol on real-world egocentric data. The trained models and code appear to be positioned for release, which would aid reproducibility. However, the significance of the empirical contribution is currently tempered by the unresolved question of whether the reported gains come from genuine whole-body understanding or from the action vector implicitly supplying the future camera trajectory.

major comments (4)
  1. [Section 3.2, Eq. (2)] The action vector is defined as the delta of root translation together with the relative rotations of all 15 upper-body joints, explicitly including the joints above the pelvis. In a head-mounted egocentric capture, the head joint rotations plus root translation determine the camera pose of the target frame. Since Eq. (2) conditions the transition P(st+1 | st,...,st-k+1, at) on this exact action, the model is handed the future camera egomotion for the frame it must generate. For largely static scenes, the next frame is almost determined by the previous frame plus the known camera motion, so the network could learn a near-geometric warp or copy solution. The paper never ablates this: Table 3 varies context length, model size, and action embedding method, but never removes or masks the head/root components. The whole-body claim is therefore not yet supported. I request an ablation that (a) removes the head and root components from the action, (b) feeds only head+root as the action, and (c) evaluates a no-action baseline, to determine how much of the gain over CDiT comes from the camera-motion shortcut.
  2. [Section 4.2, Table 1 and Section 4.3, Table 2] The baseline comparison is not controlled for the information content of the conditioning signal. CDiT is conditioned on a low-dimensional navigation trajectory (velocity and heading), while PEVA is conditioned on the full 48-dimensional pose including head orientation and all upper-body joint rotations. The reported improvements over CDiT could therefore be explained entirely by the richer camera-motion signal rather than by whole-body understanding. To support the 'whole-body conditioning' claim, the comparison should include a variant of PEVA that abandons the full pose and uses only the navigation-type signal (root translation plus perhaps heading), or a CDiT variant that receives the same full pose as input. Without this controlled comparison, the improvements in Tables 1 and 2 do not isolate the contribution of whole-body kinematics.
  3. [Section 4.1, Section 4.2, and Section 4.4] All metrics are averaged over only 5 samples per sequence, and no standard errors across seeds or significance tests are reported. The improvements over CDiT are small (e.g., LPIPS 0.303 vs. 0.313, DreamSim 0.193 vs. 0.202) and the reported error bars overlap or are simply not sufficient to establish statistical significance with n=5. I request additional sampling seeds or a paired significance test on the validation set, at least for the headline numbers in Table 1 and for the atomic-action results in Table 2.
  4. [Section 5.1 and Table 4] The planning experiment is preliminary and the paper acknowledges this, but Table 4 contains a potentially invalid presentation: for the right arm, several variance entries are negative (e.g., Shoulder Variance (0.0010, -0.0006, 0.0003), Upper Arm Variance (-0.0062, -0.0004, -0.0013)). Variances cannot be negative, so either the table reports a different statistic (e.g., covariance or raw second moments) or there is a typo. This should be corrected, as the CEM initialization described in Section 5.1 relies on these variance estimates.
minor comments (5)
  1. [Section 4.1] The training details say models predict '64-frame trajectories' but the context window is 3-15 frames and sequence-level training uses 16 frames; please clarify the relationship between these numbers.
  2. [Section 4.3 and Figure 4] The atomic actions are extracted based on thresholded positional deltas, but the thresholds themselves are not reported; please provide the exact criteria so the evaluation is reproducible.
  3. [Table 4] The formatting of the table is inconsistent (e.g., '0.004, )' in the Hand Mean row), and the left-arm statistics appear unreasonably large compared to the right-arm statistics (e.g., variance on the order of 0.1-0.25); please double-check these numbers.
  4. [Section 5.1] The planning setup says 'we only predict moving either the left or right arm' and controls 12 dimensions, but the initialization statistics in Table 4 are stated for 'arm segments' without specifying whether they are for the next action across the training dataset; please clarify.
  5. [General] The paper cites 'Rosenhahn et al., 2008' and other references in the introduction, but the reference list contains several entries with incomplete metadata (e.g., missing page numbers or venue details); a final proofread of the bibliography is needed.

Circularity Check

1 steps flagged · score 6.0 of 10

The action vector contains the target frame's egocentric camera pose (head rotation plus root translation) by construction, so the reported gains over lower-dimensional navigation-conditioned baselines may reduce to a geometric shortcut that the paper never ablates.

  1. self definitional [Section 3.1 and Section 3.2, Eq. (2); ablation study in Table 3]
    "every xj ∈ RH×W ×3 is a video frame and aj ∈ Rdact an action in the Xsens skeleton ordering (Movella, 2021) for the upper body (everything above the pelvis), representing the change in translation, together with the delta rotation of all joints relative to the previous joint rotation. ... dact = 3 + 15 × 3 = 48."

    The conditioning variable at is defined as root translation plus delta rotations of every upper-body joint, i.e., everything above the pelvis. In Nymeria, the egocentric video is captured from a head-mounted device, so the head/neck segment is an upper-body joint and its delta rotation, together with the root translation, is exactly the egocentric camera pose for the frame being predicted. The model is trained and evaluated on P(st+1 | st, ..., st−k+1, at), so the target frame's viewpoint is supplied as an input rather than inferred from whole-body action semantics. For mostly static scenes, the next frame is almost determined by the previous frame plus the supplied camera egomotion (a warp/copy solution).

full rationale

This is an empirical paper with no fitted-parameter or self-citation load-bearing derivation, so the circularity score is driven by a single construction-level leakage. The action representation is defined to include the root translation and the relative rotations of all joints above the pelvis; because the data are head-mounted egocentric captures, those exact quantities determine the camera pose of the target frame. Equation (2) conditions the next-state prediction on this action, so the target frame's viewpoint is an input by construction. The paper's ablations never test whether removing or masking the head/root action components collapses the reported gains, and the main baseline (CDiT) is conditioned only on low-dimensional navigation signals, making the comparison unable to separate 'whole-body physical understanding' from 'handed the camera trajectory'. This is partial circularity of the prediction claim, not full tautology: the model still must synthesize appearance and handle non-rigid scene changes, so the score is 6 rather than 8-10.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim is empirical, so the ledger records hand-set hyperparameters and domain assumptions on which the result depends. The most consequential item is the causal sufficiency of pose deltas, which includes the head pose that determines the egocentric camera trajectory.

free parameters (5)
  • Context window length (k) = 15 frames (default)
    Hand-chosen; ablations show monotonic improvement from 3 to 15 frames; the Markov factorization in Eq. 2 depends on this.
  • Random timeskip schedule = 16 frames sampled from a 32-second window
    The timeskip interval determines which temporal dynamics the model can learn; it is a design choice, not an empirically derived constant.
  • Action normalization ranges = translation [-1,1], rotation [-pi,pi]
    Arbitrary scaling that changes the conditioning embedding; no sensitivity analysis is provided.
  • Atomic action extraction thresholds = not reported
    Atomic actions are selected by 'thresholded positional deltas' but the thresholds are unreported, so the 100-example balanced evaluation sets are not reproducible.
  • CEM planning initialization statistics = training-set mean and variance of the next action
    Planning candidates are initialized from empirical mean and variance of the next action across the training dataset for controlled arm segments, injecting dataset statistics into the planner.
assumptions (5)
  • domain assumption Markov factorization in Eq. 2: P(s_{t+1} | s_t, ..., s_0, a_T, ..., a_0) = P(s_{t+1} | s_t, ..., s_{t-k+1}, a_t).
    The model assumes the next egocentric latent depends only on the last k latent states and one action delta, ignoring longer-range dependencies and external environment dynamics such as other agents, object motion, and lighting. Entered in Section 3.2, Eq. 2.
  • domain assumption The body pose deltas are causally sufficient controls for the egocentric visual change.
    The framework treats measured 3D pose deltas, including head pose, as the signal that determines the future view. If the camera is not rigidly coupled to the recorded body pose, or if unobserved actors affect the view, the conditioning is incomplete. Entered in Sections 1 and 3.1.
  • domain assumption Nymeria's egocentric video and Xsens motion capture are accurately synchronized and the poses are faithful.
    The training pairs are assumed to be temporally aligned and noise-free enough to learn the pose-to-vision mapping. Section 4.1.
  • ad hoc to paper Upper-body joints plus root translation represent whole-body actions.
    The 48-dimensional action includes only joints above the pelvis plus root translation; leg joint rotations are omitted. The paper still claims whole-body conditioning. Sections 3.1 and 3.2.
  • standard math Standard DDPM, transformer attention, and a fixed Stable Diffusion VAE are adequate for this latent prediction task.
    The loss in Eqs. 3 and 4 and the encoder-decoder rely on existing results from Ho et al., 2020; Nichol and Dhariwal, 2021; and Rombach et al., 2022.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Whole-Body Conditioned Egocentric Video Prediction." pith.science (2026). https://pith.science/paper/MPJM4L73

@misc{pith2026250621552,
  author       = {Pith},
  title        = {Pith review of: Whole-Body Conditioned Egocentric Video Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MPJM4L73}},
  note         = {Machine review of arXiv:2506.21552}
}
read the original abstract

We train models to Predict Ego-centric Video from human Actions (PEVA), given the past video and an action represented by the relative 3D body pose. By conditioning on kinematic pose trajectories, structured by the joint hierarchy of the body, our model learns to simulate how physical human actions shape the environment from a first-person point of view. We train an auto-regressive conditional diffusion transformer on Nymeria, a large-scale dataset of real-world egocentric video and body pose capture. We further design a hierarchical evaluation protocol with increasingly challenging tasks, enabling a comprehensive analysis of the model's embodied prediction and control abilities. Our work represents an initial attempt to tackle the challenges of modeling complex real-world environments and embodied agent behaviors with video prediction from the perspective of a human.

Figures

Figures reproduced from arXiv: 2506.21552 by the authors.

Figure 1
Figure 1. Predicting Ego-centric Video from human Actions (PEVA). Given past video frames and an action specifying a desired change in 3D pose, PEVA predicts the next video frame. Our results show that, given the first frame and a sequence of actions, our model can generate videos of atomic actions (a), simulate counterfactuals (b), and support long video generation (c). different visual outcomes depending on the environment … view at source ↗
Figure 2
Figure 2. Design of PEVA. To train on an input video, we choose a random subset of frames and encode them via a fixed encoder (a). They are then fed to a CDiT that is trained autoregressively with teacher forcing (b). During the denoising process, each token attends to same-image tokens and cross-attends to clean tokens from past image(s). Action conditioning is done via AdaLN layers. 3 PEVA In this section we describe our wh… view at source ↗
Figure 3
Figure 3. Video Quality Across Time (FID). Comparison of generation accuracy and quality as a function of time for up to 16 seconds. Qual￾itative results for 16 second rollouts can be seen in Figure 1c and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (22 more)
Figure 4
Figure 4. Figure 4: Atom Actions Generation. We include video generation examples of different atomic actions specified by 3D-body poses. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Generation Over Long-Horizons. We include 16-second video generation examples. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Planning with Counterfactuals. We demonstrate a planning example by simulating multiple action candidates using PEVA and scoring them based on their perceptual similarity to the goal, as measured by LPIPS (Zhang et al., 2018). In the first case, we show that PEVA enabl…
Figure 7
Figure 7. Figure 7: In this case, we are able to predict a sequence of actions that pulls our left arm in, similar to [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: In this case, we are able to predict a sequence of actions that lowers our left arm, but not [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: In this case, we are able to predict a sequence of actions that lowers our left arm that lowers [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: In this case, we are able to predict a sequence of actions that raises our right arm to the [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: In this case, we are able to predict a sequence of actions that moves our right arm toward [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: In this case, we are able to predict a sequence of actions that reaches toward the kettle but [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Generation Over Long-Horizons. We include 16-second video generation examples. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Generation Over Long-Horizons. We include 16-second video generation examples. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Generation Over Long-Horizons. We include 16-second video generation examples. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Generation Over Long-Horizons. We include 16-second video generation examples. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: Generation Over Long-Horizons. We include 16-second video generation examples. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: Generation Over Long-Horizons. We include 16-second video generation examples. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]
Figure 19
Figure 19. Figure 19: Generation Over Long-Horizons. We include 16-second video generation examples. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]
Figure 20
Figure 20. Figure 20: Generation Over Long-Horizons. We include 16-second video generation examples. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]
Figure 21
Figure 21. Figure 21: Generation Over Long-Horizons. We include 16-second video generation examples. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]
Figure 22
Figure 22. Figure 22: Generation Over Long-Horizons. We include 16-second video generation examples. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_22.png]
Figure 23
Figure 23. Figure 23: Generation Over Long-Horizons. We include 16-second video generation examples. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_23.png]
Figure 24
Figure 24. Figure 24: Generation Over Long-Horizons. We include 16-second video generation examples. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_24.png]
Figure 25
Figure 25. Figure 25: Generation Over Long-Horizons. We include 16-second video generation examples. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_25.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data

    cs.RO 2026-07 accept novelty 6.5 of 10

    World Action Model co-training with DINO or 3D-flow targets scales human-to-robot transfer on bimanual tasks far better than behavior cloning, while pixel prediction transfers weakly.

  2. Ego-centric Predictive Model Conditioned on Hand Trajectories

    cs.CV 2025-08 conditional novelty 6.0 of 10

    Ego-PM predicts future hand trajectories and then uses them to condition latent diffusion video generation, jointly outputting actions and future frames in egocentric and robotic scenes.

  3. Real-Time Human-Centric World Modeling for Upper-Body Human-Object Interaction

    cs.CV 2026-07 conditional novelty 5.0 of 10

    A distilled video model jointly controls multi-scale upper-body motion latents and two discrete contact states to generate real-time human–object interaction at 25 FPS.

Reference graph

Works this paper leans on

27 extracted references · 4 canonical work pages · cited by 3 Pith papers

  1. [5]

    Deep visual foresight for planning robot motion

    Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA) , pages 2786–2793. IEEE,

    work page 2017

  2. [9]

    Dream to control: Learning behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations . Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104,

    arXiv

  3. [10]

    Hierarchical world models as visual whole-body humanoid controllers

    Nicklas Hansen, Jyothir SV , Vlad Sobal, Yann LeCun, Xiaolong Wang, and Hao Su. Hierarchical world models as visual whole-body humanoid controllers. arXiv preprint arXiv:2405.18418,

    arXiv

  4. [11]

    Omnih2o: Universal and dexterous human-to-humanoid whole-body teleoperation and learning

    Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Omnih2o: Universal and dexterous human-to-humanoid whole-body teleoperation and learning. arXiv preprint arXiv:2406.08858, 2024a. Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Learning human-to-...

    arXiv 2024

  5. [13]

    Animate anyone: Consistent and controllable image-to-video synthesis for character animation

    Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint arXiv:2311.17117,

    arXiv

  6. [15]

    Multi-task interactive robot fleet learning with visual world models

    Huihan Liu, Yu Zhang, Vaarij Betala, Evan Zhang, James Liu, Crystal Ding, and Yuke Zhu. Multi-task interactive robot fleet learning with visual world models. arXiv preprint arXiv:2410.22689,

    arXiv

  7. [16]

    Nymeria: A massive collection of multimodal egocentric daily motion in the wild

    15 Qianli Ma et al. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. arXiv preprint arXiv:2406.09905,

    arXiv

  8. [17]

    Vip: Towards universal visual reward and representation via value-implicit pre-training

    Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00030,

    arXiv

Show all 27 references
  1. [19]

    Learning humanoid locomotion over challenging terrain

    Ilija Radosavovic, Sarthak Kamat, Trevor Darrell, and Jitendra Malik. Learning humanoid locomotion over challenging terrain. arXiv preprint arXiv:2410.03654,

    arXiv

  2. [21]

    Nomad: Goal masked diffusion policies for navigation and exploration

    Ajay Sridhar, Dhruv Shah, Catherine Glossop, and Sergey Levine. Nomad: Goal masked diffusion policies for navigation and exploration. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 63–70. IEEE,

    2024

  3. [22]

    Human motion diffusion model

    Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209.14916,

    arXiv

  4. [23]

    Magicanimate: Temporally consistent human image animation using diffusion model

    Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. arXiv preprint arXiv:2311.16498,

    arXiv

  5. [24]

    Learning interactive real-world simulators

    Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114, 1(2):6,

    arXiv

  6. [25]

    Video as the new language for real-world decision making

    16 Sherry Yang, Jacob Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, Andre Barreto, Pieter Abbeel, and Dale Schuurmans. Video as the new language for real-world decision making. arXiv preprint arXiv:2402.17139,

    arXiv

  7. [26]

    Egobody: Human body shape and motion of interacting people from head-mounted devices

    Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Taein Kwon, Marc Pollefeys, Federica Bogo, and Siyu Tang. Egobody: Human body shape and motion of interacting people from head-mounted devices. arXiv preprint arXiv:2204.06953,

    arXiv

  8. [27]

    Dino-wm: World models on pre-trained visual features enable zero-shot planning

    Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983,

    arXiv

  9. [1987]

    A path towards autonomous machine intelligence version 0.9

    Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1): 1–62,

    2022

  10. [1997]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792,

    arXiv

  11. [2015]

    Dreamsim: Learning new dimensions of human visual similarity using synthetic data

    Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. arXiv preprint arXiv:2306.09344,

    arXiv

  12. [2016]

    Diamond: Diffusion as a model of environment dreams

    Eloi Alonso et al. Diamond: Diffusion as a model of environment dreams. arXiv preprint arXiv:2401.02644,

    arXiv

  13. [2017]

    Learning visual predictive models of physics for playing billiards

    Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning visual predictive models of physics for playing billiards. arXiv preprint arXiv:1511.07404,

    arXiv

  14. [2018]

    Visual foresight: Model-based deep reinforcement learning for vision-based robotic control

    Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00568,

    arXiv

  15. [2020]

    Egolm: Multi-modal language model of egocentric motions

    Fangzhou Hong, Vladimir Guzov, Hyo Jin Kim, Yuting Ye, Richard Newcombe, Ziwei Liu, and Lingni Ma. Egolm: Multi-modal language model of egocentric motions. arXiv preprint arXiv:2409.18127,

    arXiv

  16. [2021]

    Gr00t n1: An open foundation model for generalist humanoid robots

    J Bjorck Nvidia, F Castaneda, N Cherniadev, X Da, R Ding, L Fan, Y Fang, D Fox, F Hu, S Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734,

    arXiv

  17. [2022]

    World models

    David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122,

    arXiv

  18. [2023]

    V-jepa 2: Self-supervised video models enable under- standing, prediction and planning

    Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable under- standing, prediction and planning. arXiv preprint arXiv:2506.09985,

    arXiv

  19. [2024]

    Expressive whole-body control for humanoid robots

    Xuxin Cheng, Yandong Ji, Junming Chen, Ruihan Yang, Ge Yang, and Xiaolong Wang. Expressive whole-body control for humanoid robots. arXiv preprint arXiv:2402.16796,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.