Pith. sign in

REVIEW 5 major objections 6 minor 87 references

MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction

T0 review · 5 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a retrospective 3D tracking model, once its future frames are masked and a small adapter is trained, becomes an accurate prospective 3D motion forecaster.

desk verdict Genuinely new formulation, but the headline numbers only measure how well the model imitates TrackCraft3R, not true future 3D motion. read the letter →

arxiv 2607.16192 v1 pith:PZUXLZY5 submitted 2026-07-17 cs.CV

Homanga Bharadhwaj , Yash Jangir This is my paper
classification cs.CV
keywords future3Dsceneflowvideopredictionpriorspointtrackforecastinghuman-objectinteractionreference-anchoredtracksmasklatentsmodelfine-tuningembodiedintelligence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MotionForesight tries to establish that the temporal priors inside pretrained video models can be redirected—without retraining the large model—into explicit forecasts of future 3D motion. Given seven observed RGB frames of a human manipulating an object, the method predicts where points on that object will be for the next fifteen frames, expressed as reference-anchored 3D trajectories. The key move is to take a tracker that normally reconstructs motion from a complete clip, hide the future frames behind learned mask latents, and train only a thin adapter so the model extrapolates instead of retrospects. The paper argues this works with just 40,000 monocular human-interaction videos and no language or action labels, and that it beats much larger, language-grounded models trained on over a million videos. A sympathetic reader would care because it suggests video priors can be cheaply converted into actionable geometric predictions for embodied agents.

What carries the argument

The central mechanism is the mask latent: a single learned vector that fills the context stream at future timestamps in place of unavailable RGB and pointmap encodings, while temporal Rotary Position Embedding keeps time indices distinct. The frozen video diffusion transformer, together with a frozen tracking adapter, acts as a single-step latent regressor; a fresh low-rank adapter and a patch/head pair turn the retrospective tracker into a forecaster. The reference-frame query latent is repeated across all time slots, so every output slot answers the same geometric question: where is each reference point at time t?

What would settle it

Render synthetic videos with known ground-truth 3D trajectories from a physics simulator, or capture real scenes with motion capture or depth sensors, then run the method on the observed prefix and compare against the true future tracks; if the error on such metric ground truth is far worse than on the pseudo-labels, the bottleneck is the pseudo-label pipeline rather than the forecasting model.

Watch

Extended reading notes

Core claim

Convert a retrospective dense 3D tracker into a forward predictor by replacing future RGB and pointmap latents with learned mask latents, keeping the video backbone and tracking decoder frozen, and training only a fresh low-rank adapter plus a patch embedding, output head, and mask latents (12.19M trainable parameters total). The model observes 7 frames, predicts 15 steps of metric 3D tracks anchored to reference-frame object points, expressed in the last-observed camera frame. Pseudo-ground-truth tracks are generated offline from complete clips. The discovered claim is that this simple re-purposing, trained on 40K human-interaction videos without language, outperforms a concurrent sparse-po

Load-bearing premise

The offline pipeline that estimates depth, camera motion, and dense 3D tracks from complete clips produces 3D trajectories accurate enough to serve as ground truth for both training and evaluation; if it drifts or hallucinates, the reported numbers measure fidelity to that tracker's output, not true 3D motion.

Editorial extensions

If this is right

  • Future 3D object motion can be predicted directly from image context, bypassing pixel generation and tracking-after-generation.
  • The same representation covers rigid, articulated, and deformable motion because it predicts dense point trajectories rather than rigid poses.
  • Scaling from 1K to 40K human-interaction videos improves most motion-quality metrics, suggesting the recipe benefits from more diverse interactions.
  • Even without language or action labels, the model beats a larger model that receives ground-truth action descriptions, indicating that geometric grounding can outweigh auxiliary language supervision.
  • The frozen video prior transfers to out-of-distribution phone videos of unseen objects and scenes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the central claim holds, the same mask-latent trick could be applied to other retrospective spatio-temporal models, such as video segmentation or 3D reconstruction, to turn them into forecasters, provided they have a fixed query interface.
  • A testable extension is to make the forecaster stochastic: sample multiple mask latents or inject noise to produce a distribution of plausible futures, then evaluate with distance-to-set metrics rather than single-trajectory error.
  • The reported under-prediction of motion magnitude (motion ratio 0.72) suggests a systematic bias toward conservative forecasts; adding a velocity-magnitude penalty to the loss could correct this without architectural changes.
  • Because pseudo-ground-truth is derived from a tracker, the method's ceiling is likely bound by that tracker; replacing it with a more accurate 3D tracker or jointly learning geometry could lift both training signal and evaluation fidelity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes MotionForesight, a method that re-purposes the retrospective dense 3D tracker TrackCraft3R (built on the Wan2.1 video DiT) into a future 3D scene-flow forecaster. Observed RGB frames and estimated pointmaps are encoded for the first T1 timesteps; future slots are filled with learned mask latents, and a small trainable adapter (~12.19M parameters) predicts reference-anchored future 3D tracks. Training supervision is pseudo-ground-truth extracted offline by TrackCraft3R from complete SSv2 clips. The method is evaluated on held-out SSv2 clips and 50 OOD phone videos against MolmoMotion and video-generation-plus-tracking baselines, with standard ADE/FDE/PWT metrics plus newly introduced motion-conditional diagnostics (TVO, VVO, MoveF1, MoveIoU, DQS). The central claim is that a frozen video prior plus a small adapter, trained on 40K videos without language or action labels, generalizes OOD and outperforms much larger language-conditioned models.

Significance. If the claims hold, the paper makes a valuable practical and conceptual contribution: it shows that large retrospective video/tracking models can be efficiently turned into forward geometric predictors with a modest amount of data and parameters, and it introduces motion-conditional diagnostics that are more informative than displacement-only metrics for a multimodal forecasting task. The authors are transparent about the pseudo-GT limitation in Section 5, and the OOD phone-video evaluation is a useful stress test. However, the verification of the central quantitative claims is weakened by the dependence of both training and evaluation on the same pseudo-GT pipeline, the absence of trivial kinematic baselines, and the lack of error bars. These issues are fixable but need to be addressed before the claims of superiority and generalization are fully supported.

major comments (5)
  1. [§3.1, §A.6, Eqs. (9)–(10)] The training labels and the evaluation targets are produced by the same offline pipeline: TrackCraft3R run on complete clips, with observed pointmaps from DepthAnything3 and the same camera/mask preprocessing. Since the model is trained to regress exactly these tracker outputs and the OOD phone-video metrics are also computed against the same tracker (§A.6), the reported ADE/FDE and the cross-domain generalization numbers measure fidelity to TrackCraft3R's retrospective output, not necessarily true future 3D motion. The paper itself acknowledges this in Section 5. This is the central load-bearing issue. I would like to see at least one of: (a) evaluation on independent metric 3D ground truth (e.g., RGB-D or mocap-captured interactions), (b) evaluation against pseudo-GT from a different tracker/depth pipeline to quantify cross-pipeline agreement, or (c) an explicit error analysis showing
  2. [Tables 1–4] No trivial baselines are reported. A zero-motion baseline (all predicted points remain at their last observed position) and a constant-velocity baseline (extrapolate the observed 3D velocity of each query point) are cheap, standard, and essential for interpreting the absolute ADE/FDE values. Without these, the relative gains over MolmoMotion and video-generation baselines do not establish that the model has learned meaningful extrapolation beyond the observed dynamics, rather than, for example, a conservative near-zero-motion prior. The motion-conditional diagnostics in Table 4 (especially the motion ratio r̄=0.72) already suggest under-prediction, which makes this baseline gap more acute.
  3. [Table 3] The 'scaling with human-interaction videos' claim is not consistently supported by the numbers. The 40K model has worse OOD ADE/FDE (9.31/14.88) than the 10K model (8.97/14.63), and the gains from 1K to 40K on SSv2 are small (ADE 4.81→4.47). The paper attributes this to multimodality and a single recorded future, but this explanation is post hoc. Given that scaling is one of the four research questions, the experiment needs either multiple seeds with confidence intervals, an alternative metric that explicitly rewards plausible alternative futures, or a clear statement of which specific claim the scaling experiment is designed to support and what evidence would falsify it.
  4. [§4.2, Table 1] Several headline comparisons are within what appears to be noise, yet no uncertainty quantification is reported. For example, on OOD phone videos, MotionForesight's ADE is 9.31 vs. MolmoMotion's 9.50 (a 2% difference) and PWT is 54 vs. 53; on SSv2, the PWT difference is 76 vs. 70 over 150 clips. Without standard errors, confidence intervals, or multiple evaluation seeds, the claim that MotionForesight 'outperforms substantially larger models' is not strongly supported. Please report variability across evaluation subsets or training runs.
  5. [§3.4, Table 5] The paper's 'key insight' is that pretrained video models encode motion priors that can be redirected. However, there is no ablation that isolates the contribution of the frozen video prior. A proper control would train the same adapter and mask latents on a randomly initialized video transformer (or a non-pretrained transformer with the same architecture), and ideally also compare to fine-tuning the base model. Without such a control, the improved performance could come from the overall architecture, the training recipe, or the pseudo-GT data rather than from the specific pretrained video prior that the paper emphasizes.
minor comments (6)
  1. [§4.2] Typo: 'proviledged' should be 'privileged'.
  2. [Table 4 caption] The note 'Darker and lighter blue denote the best and second-best learned result' cannot be read in monochrome print or for color-blind readers; use symbols or boldface.
  3. [§4.1 / Table 1] The text defines PWT as 'percentage of predictions within a fixed distance threshold' and later mentions '@5cm', but Table 1 header just says 'PWT'. Make the threshold explicit in every table.
  4. [Eq. (7)] The denominators include an ε term but ε is not defined in the main text or supplement. Please define it or remove it for clarity.
  5. [§4.1 / §A.6] The evaluation sizes (150 SSv2 clips, 50 phone videos) are small. A brief sentence stating the standard error of the mean for the headline metrics would help calibrate the reader's confidence.
  6. [§3.1] The mask extraction uses the object name from the dataset annotation. The paper says text is used only for preprocessing, but this detail is worth restating in the main text to avoid any impression that language is available at inference.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline ADE/FDE are computed against the same TrackCraft3R pseudo-GT used as training targets, so the forecasting numbers measure fidelity to that tracker, not independent 3D motion.

  1. fitted input called prediction [§3.1 (Eq. 7), §A.6 (Eq. 9), §5]
    "Pseudo-ground-truth future tracks are generated by running dense 3D tracking on complete human interaction clips, then training the model with those future observations removed. ... For held-out validation clips with pseudo-ground-truth tracks, we evaluate future average displacement error and final displacement error over valid object query points. ... Supervision is also pseudo ground truth, so errors from monocular depth, camera estimation, segmentation, and tracking can propagate into both training and evaluation."

    Eq. 7's future term is exactly the sum over t>=T1 of v_tq ||Xhat_t(q)-X_t(q)||^2, and Eq. 9 (ADE) evaluates the same quantity over the same pseudo-GT X_t. Both X_t sources are TrackCraft3R tracks extracted from complete clips (§3.1, §A.6). Thus the reported ADE/FDE is, by construction, agreement with the retrospective tracker used as the training target, not with independently measured future 3D motion. Because MotionForesight is a masked-future modification of the same frozen tracker, and observed pointmaps come from the same monocular depth/camera pipeline that feeds TrackCraft3R, systematic depth, camera, and tracking biases enter training and evaluation identically. This is a partial circularity, explicitly conceded in §5.

full rationale

The central quantitative claim—that a 40k-video, 12.19M-parameter model forecasts accurate future metric 3D motion—is not independently anchored. The training objective (Eq. 7) and the evaluation metrics (Eqs. 9–10) both use X_t(q), the 'pseudo-ground-truth' tracks produced by TrackCraft3R from complete clips. The future term of Eq. 7 is identical in form to the ADE of Eq. 9, so low error means the model reproduces TrackCraft3R's retrospective tracks from an observed prefix, rather than matching an external ground truth. The model is built inside TrackCraft3R (frozen video DiT, frozen tracking LoRA, frozen track decoder) and its observed pointmaps come from the same DepthAnything3-camera pipeline that feeds the tracker, so systematic metric-scale, occlusion, and hallucination errors enter both sides of the benchmark in the same way. Section 5 concedes that 'errors from monocular depth, camera estimation, segmentation, and tracking can propagate into both training and evaluation.' This is a genuine partial circularity: comparisons to MolmoMotion and videogen baselines are fair in the narrow sense that all methods are scored against the same pseudo-GT, but the headline numbers measure fidelity to the tracker, not to true 3D motion. There is no load-bearing self-citation chain, uniqueness theorem, or ansatz-smuggled-via-citation; references to the authors' prior work are contextual. The architecture-ablation and scaling results retain independent content conditional on the chosen pseudo-GT, which is why the score is 6 rather than higher: the central quantitative evaluation reduces to agreement with a single pseudo-label generator shared by training and evaluation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on pseudo-GT labels and on the transferability of frozen video/tracking priors; these are domain assumptions rather than mathematical axioms. The main hand-chosen numbers are the task horizon, loss weights, and the motion threshold used to define moving points. No new physical entities are introduced; the learned mask latents are model parameters, not independent postulates.

free parameters (3)
  • motion threshold τ = 2 cm (τ∈{1,5,10} also checked)
    Hand-chosen in §4.5/Appendix B to define 'moving points' for TVO/VVO/MoveF1/DQS; determines the 108/150 clip subset in Table 4 and all motion-conditional numbers.
  • loss weights λ_obs, λ_fut = 0.25 / 1.0, coordinate MSE scaled ×10
    Chosen by hand in §3.5 and §A.4; directly sets the balance between preserving the tracking interface and learning future extrapolation, affecting every reported result.
  • task horizon T1=7, T2=15 = 7 observed, 15 predicted, 22 total
    The default setting in §3; all ADE/FDE/PWT numbers are specific to this short-horizon forecasting task.
assumptions (4)
  • domain assumption Pseudo-GT tracks from TrackCraft3R over complete clips (with DepthAnything3, camera estimation, and SAM masks) are accurate proxies for true 3D object motion.
    Invoked in §3.1 and §A.6; both training supervision and evaluation metrics rely on this assumption. If false, reported errors do not measure real 3D forecast accuracy.
  • domain assumption Frozen video priors in Wan2.1 and TrackCraft3R contain transferable motion knowledge sufficient for a low-rank adapter to forecast future 3D tracks.
    This is the paper's central motivating hypothesis (§1, §3.4); the experiments test it, but the architecture assumes it a priori by freezing the large components.
  • ad hoc to paper Shared learned mask latents for future RGB and pointmap slots, combined with temporal RoPE, are a sufficient representation of 'unknown future' context.
    Introduced in §3.4 Eq. 6; the model's ability to forecast depends on this chosen encoding of missing future observations.
  • domain assumption SSv2 object-name cues can identify and segment the manipulated object, and action-template stratification provides useful diversity.
    Used in the offline curation pipeline (§3.1) and the scaling experiment (§4.3); failures here would corrupt both pseudo-GT generation and the data-scaling conclusions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction." pith.science (2026). https://pith.science/paper/PZUXLZY5

@misc{pith2026260716192,
  author       = {Pith},
  title        = {Pith review of: MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PZUXLZY5}},
  note         = {Machine review of arXiv:2607.16192}
}
read the original abstract

Humans can infer how objects are likely to move from passive observation: a cup may be lifted, a drawer may slide, and a lid may rotate shut. Such predictions expose the physical consequences of interaction needed to act in the real world. We study how to learn this anticipation from ordinary monocular videos of human-object interaction. Given a short observed video context, MotionForesight predicts future 3D trajectories for points on the manipulated object. This casts interaction prediction as object-centered 3D motion forecasting without any assumptions on the object properties. Our key insight is that video prediction models already encode rich priors about how objects move during human interactions. We redirect these priors from pixel prediction toward future 3D scene flow. We start from a dense 3D tracker built on a pretrained video model, generate pseudo-ground-truth tracks from complete clips, and train the forecaster using only the observed frames. We replace future RGB and geometry with learned mask latents and train a lightweight adapter to turn the retrospective tracking representation into a forward predictor, while freezing the large video and tracking components. Using just 40k human videos and no auxiliary inputs such as language, MotionForesight generalizes across diverse out-of-distribution objects, environments, viewpoints, and interactions. It also outperforms substantially larger models that use over a million training videos. These results show that we can efficiently re-purpose video priors into explicit geometric forecasts for embodied intelligence. https://motionforesight.github.io/

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

87 extracted references · 5 canonical work pages

  1. [1]

    Gibson.The Ecological Approach to Visual Perception

    James J. Gibson.The Ecological Approach to Visual Perception. Houghton Mifflin, Boston,

  2. [2]

    MIT Press, Cambridge, MA, 1979

    Shimon Ullman.The Interpretation of Visual Motion. MIT Press, Cambridge, MA, 1979. ISBN 9780262210072

    1979

  3. [3]

    Battaglia, Jessica B

    Peter W. Battaglia, Jessica B. Hamrick, and Joshua B. Tenenbaum. Simulation as an engine of physical scene understanding.Proceedings of the National Academy of Sciences, 110(45): 18327–18332, 2013. doi: 10.1073/pnas.1306572110

    doi:10.1073/pnas.1306572110 2013

  4. [4]

    Rational imitation in preverbal infants

    György Gergely, Harold Bekkering, and Ildikó Király. Rational imitation in preverbal infants. Nature, 415(6873):755, 2002. doi: 10.1038/415755a

    work page doi:10.1038/415755a 2002

  5. [5]

    MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction

    Jianing Zhang, Chenhao Zheng, Yajun Yang, Max Argus, Rustin Soraki, Winson Han, Taira Anderson, Chun-Liang Li, Shuo Liu, Jiafei Duan, Zhongzheng Ren, Jieyu Zhang, and Ranjay Krishna. MolmoMotion: Forecasting point trajectories in 3d with language instruction.arXiv preprint arXiv:2606.18558, 2026. doi: 10.48550/arXiv.2606.18558

    work page Pith review arXiv doi:10.48550/arxiv.2606.18558 2026

  6. [6]

    Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Vision

    Tomoya Yoshida, Shuhei Kurita, Taichi Nishimura, and Shinsuke Mori. Generating 6dof object manipulation trajectories from action description in egocentric vision.arXiv preprint arXiv:2506.03605, 2025. doi: 10.48550/arXiv.2506.03605

    work page Pith review arXiv doi:10.48550/arxiv.2506.03605 2025

  7. [7]

    ObjectForesight: Predicting future 3d object trajectories from human videos.arXiv preprint arXiv:2601.05237,

    Rustin Soraki, Homanga Bharadhwaj, Ali Farhadi, and Roozbeh Mottaghi. ObjectForesight: Predicting future 3d object trajectories from human videos.arXiv preprint arXiv:2601.05237,

    arXiv

  8. [8]

    Track2Act: Predicting point tracks from internet videos enables generalizable robot manipulation

    Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2Act: Predicting point tracks from internet videos enables generalizable robot manipulation. In Computer Vision – ECCV 2024, volume 15134 ofLecture Notes in Computer Science, pages 306–324. Springer, 2024. doi: 10.1007/978-3-031-73116-7_18

    doi:10.1007/978-3-031-73116-7_18 2024

Show all 87 references
  1. [9]

    Any-point trajectory modeling for policy learning

    Chuan Wen, Xingyu Lin, John Ian Reyes So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. InRobotics: Science and Systems (RSS),

  2. [10]

    Motion tracks: A unified representation for human-robot transfer in few-shot imitation learning

    Juntao Ren, Priya Sundaresan, Dorsa Sadigh, Sanjiban Choudhury, and Jeannette Bohg. Motion tracks: A unified representation for human-robot transfer in few-shot imitation learning. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 8802–8810. IEEE, 2025

    2025

  3. [11]

    TrackCraft3R: Repurposing video diffusion transformers for dense 3d tracking.arXiv preprint arXiv:2605.12587, 2026

    Jisu Nam, Jahyeok Koo, Soowon Son, Jaewoo Jung, Honggyu An, Junhwa Hur, and Seungryong Kim. TrackCraft3R: Repurposing video diffusion transformers for dense 3d tracking.arXiv preprint arXiv:2605.12587, 2026. doi: 10.48550/arXiv.2605.12587

    2026 doi arXiv

  4. [12]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314,

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314,

    arXiv

  5. [13]

    World models.arXiv preprint arXiv:1803.10122, 2018

    David Ha and Jürgen Schmidhuber. World models.arXiv preprint arXiv:1803.10122, 2018. doi: 10.48550/arXiv.1803.10122

    2018 doi arXiv

  6. [14]

    Deep visual foresight for planning robot motion

    Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2786–2793. IEEE,

  7. [15]

    Decomposing motion and content for natural video sequence prediction

    Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. InInternational Conference on Learning Representations (ICLR), 2017

    2017

  8. [16]

    An uncertain future: Forecasting from static images using variational autoencoders

    Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autoencoders. InProceedings of the European Conference on Computer Vision (ECCV), 2016

    2016

  9. [17]

    Gen2Act: Human video generation in novel scenarios enables generalizable robot manipulation

    Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani. Gen2Act: Human video generation in novel scenarios enables generalizable robot manipulation. In Joseph Lim, Shuran Song, and Hae...

  10. [18]

    Video prediction policy: A generalist robot policy with predictive visual representations

    Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-...

    2025

  11. [19]

    Video generators are robot policies.arXiv preprint arXiv:2508.00795, 2025

    Junbang Liang, Pavel Tokmakov, Ruoshi Liu, Sruthi Sudhakar, Paarth Shah, Rares Ambrus, and Carl Vondrick. Video generators are robot policies.arXiv preprint arXiv:2508.00795, 2025. doi: 10.48550/arXiv.2508.00795

    2025 doi arXiv

  12. [20]

    Learning latent action world models in the wild.arXiv preprint arXiv:2601.05230,

    Quentin Garrido, Tushar Nagarajan, Basile Terver, Nicolas Ballas, Yann LeCun, and Michael Rabbat. Learning latent action world models in the wild.arXiv preprint arXiv:2601.05230,

  13. [21]

    V-JEPA 2: Self-supervised video models enable understanding, prediction and planning

    Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, et al. V-JEPA 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025. doi: 10.48550/arXiv.2506.09985

    2025 doi arXiv

  14. [22]

    URLhttps://proceedings.mlr.press/v305/bharadhwaj25a.html

  15. [23]

    Contrastive learning of structured world models

    Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. InInternational Conference on Learning Representations (ICLR), 2020

    2020

  16. [24]

    SomethingSomething

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, FlorianHoppe, ChristianThurau, IngoBax, andRolandMemisevic. The“SomethingSomething” video database for...

    2017

  17. [25]

    Doell, and Jason J

    Pradipto Das, Chenliang Xu, Richard F. Doell, and Jason J. Corso. A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013

    2013

  18. [26]

    doi: 10.48550/arXiv.2601.05230

    doi

  19. [27]

    Yin Li, Miao Liu, and James M. Rehg. In the eye of beholder: Joint learning of gaze and actions in first person video. InProceedings of the European Conference on Computer Vision (ECCV), 2018

    2018

  20. [28]

    Dream to con- trol: Learning behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to con- trol: Learning behaviors by latent imagination. InInternational Conference on Learning Representations (ICLR), 2020

    2020

  21. [29]

    Miao Liu, Siyu Tang, Yin Li, and James M. Rehg. Forecasting human-object interaction: Joint prediction of motor attention and actions in first person video. InComputer Vision – ECCV 2020, Lecture Notes in Computer Science, pages 704–721. Springer, 2020. doi: 10.1007/ 978-3-030...

    2020

  22. [30]

    Human–object interaction prediction in videos through gaze following.Computer Vision and Image Understanding, 233: 103741, 2023

    Zhifan Ni, Esteve Valls Mascaró, Hyemin Ahn, and Dongheui Lee. Human–object interaction prediction in videos through gaze following.Computer Vision and Image Understanding, 233: 103741, 2023. ISSN 1077-3142. doi: 10.1016/j.cviu.2023.103741. 14 MotionForesight Brains, Bots, and...

    2023

  23. [31]

    ContactGrasp: Functional multi-finger grasp synthesis from contact

    Samarth Brahmbhatt, Ankur Handa, James Hays, and Dieter Fox. ContactGrasp: Functional multi-finger grasp synthesis from contact. InIEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019

    2019

  24. [32]

    Scaling egocentric vision: The EPIC-KITCHENS dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The EPIC-KITCHENS dataset. InProceedings of the European Conference...

    2018

  25. [33]

    Joint hand motion and interaction hotspots prediction from egocentric videos

    Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  26. [34]

    Ego4D: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4D: Around the world in 3,000 hours of egocentric video. InProceedings of the IEEE/CVF Conference on Computer Vision a...

    2022

  27. [35]

    Newtonian image understanding: Unfolding the dynamics of objects in static images

    Roozbeh Mottaghi, Hessam Bagherinezhad, Mohammad Rastegari, and Ali Farhadi. Newtonian image understanding: Unfolding the dynamics of objects in static images. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

    2016

  28. [36]

    Grounded human-object interaction hotspots from video

    Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman. Grounded human-object interaction hotspots from video. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8688–8697, 2019

    2019

  29. [37]

    Dandan Shan, Jiaqi Geng, Michelle Shu, and David F. Fouhey. Understanding human hands in contact at internet scale. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020

    2020

  30. [38]

    Human hands as probes for interactive object understanding

    Mohit Goyal, Sahil Modi, Rishabh Goyal, and Saurabh Gupta. Human hands as probes for interactive object understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  31. [39]

    Hand- sOnVLM: Vision-language models for hand-object interaction prediction.arXiv preprint arXiv:2412.13187, 2024

    Chen Bao, Jiarui Xu, Xiaolong Wang, Abhinav Gupta, and Homanga Bharadhwaj. Hand- sOnVLM: Vision-language models for hand-object interaction prediction.arXiv preprint arXiv:2412.13187, 2024. doi: 10.48550/arXiv.2412.13187

    2024 doi arXiv

  32. [40]

    Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani

    Kaichun Mo, Leonidas J. Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2Act: From pixels to actions for articulated 3d objects. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021

    2021

  33. [41]

    Mask2Act: Predictive multi-object tracking as video pre- training for robot manipulation

    Junbo Zhang and Kaisheng Ma. Mask2Act: Predictive multi-object tracking as video pre- training for robot manipulation. In36th British Machine Vision Conference 2025, BMVC 2025. BMVA, 2025. URLhttps://bmvc2025.bmva.org/proceedings/124/

    2025

  34. [42]

    3d hand shape and pose estimation from a single RGB image

    Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, and Junsong Yuan. 3d hand shape and pose estimation from a single RGB image. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019

    2019

  35. [43]

    Black, Ivan Laptev, and Cordelia Schmid

    Yana Hasson, Gül Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 15 ...

    2019

  36. [44]

    Guide to the carnegie mellon university multimodal activity (CMU- MMAC) database

    Fernando De la Torre, Jessica Hodgins, Adam Bargteil, Xavier Martin, Justin Macey, Alex Collado, and Pep Beltran. Guide to the carnegie mellon university multimodal activity (CMU- MMAC) database. Technical report, Carnegie Mellon University, 2008

    2008

  37. [45]

    FrankMocap: Fast monocular 3d hand and body motion capture by regression and integration.arXiv preprint arXiv:2008.08324, 2020

    Yu Rong, Takaaki Shiratori, and Hanbyul Joo. FrankMocap: Fast monocular 3d hand and body motion capture by regression and integration.arXiv preprint arXiv:2008.08324, 2020. doi: 10.48550/arXiv.2008.08324

    2008 doi arXiv

  38. [46]

    Flowing from reasoning to motion: Learning 3d hand trajectory prediction from egocentric human interaction videos.arXiv preprint arXiv:2512.16907, 2025

    Mingfei Chen, Yifan Wang, Zhengqin Li, Homanga Bharadhwaj, Yujin Chen, Chuan Qin, Ziyi Kou, Yuan Tian, Eric Whitmire, Rajinder Sodhi, Hrvoje Benko, Eli Shlizerman, and Yue Liu. Flowing from reasoning to motion: Learning 3d hand trajectory prediction from egocentric human inter...

    2025 doi

  39. [47]

    PVN3D: A deep point-wise 3d keypoints voting network for 6dof pose estimation

    Yisheng He, Wei Sun, Haibin Huang, Jianran Liu, Haoqiang Fan, and Jian Sun. PVN3D: A deep point-wise 3d keypoints voting network for 6dof pose estimation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020

    2020

  40. [48]

    Segmentation-driven 6d object pose estimation

    Yinlin Hu, Joachim Hugonot, Pascal Fua, and Mathieu Salzmann. Segmentation-driven 6d object pose estimation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019

    2019

  41. [49]

    SSD-6D: Making RGB-based 3d detection and 6d pose estimation great again

    Wadim Kehl, Fabian Manhardt, Federico Tombari, Slobodan Ilic, and Nassir Navab. SSD-6D: Making RGB-based 3d detection and 6d pose estimation great again. InProceedings of the IEEE International Conference on Computer Vision (ICCV), 2017

    2017

  42. [50]

    Hand pose estimation via latent 2.5d heatmap regression

    Umar Iqbal, Pavlo Molchanov, Thomas Breuel, Juergen Gall, and Jan Kautz. Hand pose estimation via latent 2.5d heatmap regression. InProceedings of the European Conference on Computer Vision (ECCV), 2018

    2018

  43. [51]

    SAM 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, et al. SAM 2: Segment anything in images and videos. InInternational Conference on Learning Representa- tions (ICLR), 2025

    2025

  44. [52]

    Learning to estimate 3d hand pose from single RGB images

    Christian Zimmermann and Thomas Brox. Learning to estimate 3d hand pose from single RGB images. InProceedings of the IEEE International Conference on Computer Vision (ICCV), 2017

    2017

  45. [53]

    Liang, Alexander Sax, et al

    SAM 3D Team, Xingyu Chen, Fu-Jen Chu, Pierre Gleize, Kevin J. Liang, Alexander Sax, et al. SAM 3D: 3dfy anything in images.arXiv preprint arXiv:2511.16624, 2025. doi: 10.48550/arXiv.2511.16624

    2025 doi arXiv

  46. [54]

    Self-supervised point cloud prediction using 3d spatio-temporal convolutional networks

    Benedikt Mersch, Xieyuanli Chen, Jens Behley, and Cyrill Stachniss. Self-supervised point cloud prediction using 3d spatio-temporal convolutional networks. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors,Proceedings of the 5th Conference on Robot Learning, volume ...

  47. [55]

    Visual point cloud forecasting enables scalable autonomous driving

    Zetong Yang, Li Chen, Yanan Sun, and Hongyang Li. Visual point cloud forecasting enables scalable autonomous driving. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14673–14684, June 2024. 16 MotionForesight Brains, Bots, and ...

    2024

  48. [56]

    PoseCNN: A convo- lutional neural network for 6d object pose estimation in cluttered scenes

    Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. PoseCNN: A convo- lutional neural network for 6d object pose estimation in cluttered scenes. InProceedings of Robotics: Science and Systems, 2018. doi: 10.15607/RSS.2018.XIV.019

    2018 doi

  49. [57]

    ManipTrans: Efficient dexterousbimanualmanipulationtransferviaresiduallearning

    Kailin Li, Puhao Li, Tengyu Liu, Yuyang Li, and Siyuan Huang. ManipTrans: Efficient dexterousbimanualmanipulationtransferviaresiduallearning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6991–7003, 2025

    2025

  50. [58]

    Structured 3d latents for scalable and versatile 3d generation

    Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  51. [59]

    World models for learn- ing dexterous hand-object interactions from human videos.arXiv preprint arXiv:2512.13644,

    Raktim Gautam Goswami, Amir Bar, David Fan, Tsung-Yen Yang, Gaoyue Zhou, Prashanth Krishnamurthy, Michael Rabbat, Farshad Khorrami, and Yann LeCun. World models for learn- ing dexterous hand-object interactions from human videos.arXiv preprint arXiv:2512.13644,

  52. [60]

    Qi, and Leonidas J

    Xingyu Liu, Charles R. Qi, and Leonidas J. Guibas. FlowNet3D: Learning scene flow in 3d point clouds. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 529–537, June 2019. doi: 10.1109/CVPR.2019.00062

    2019

  53. [62]

    Harley, Yang You, Xinglong Sun, Yang Zheng, Nikhil Raghuraman, Yunqi Gu, Sheldon Liang, Wen-Hsuan Chu, Achal Dave, Suya You, Rares Ambrus, Katerina Fragkiadaki, and Leonidas J

    Adam W. Harley, Yang You, Xinglong Sun, Yang Zheng, Nikhil Raghuraman, Yunqi Gu, Sheldon Liang, Wen-Hsuan Chu, Achal Dave, Suya You, Rares Ambrus, Katerina Fragkiadaki, and Leonidas J. Guibas. AllTracker: Efficient dense point tracking at high resolution. In Proceedings of the...

    2025

  54. [63]

    ViP3D: End-to-end visual trajectory prediction via 3d agent queries

    Junru Gu, Chenxu Hu, Tianyuan Zhang, Xuanyao Chen, Yilun Wang, Yue Wang, and Hang Zhao. ViP3D: End-to-end visual trajectory prediction via 3d agent queries. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  55. [64]

    DELTA: Dense efficient long-range 3d tracking for any video

    Tuan Duc Ngo, Peiye Zhuang, Chuang Gan, Evangelos Kalogerakis, Sergey Tulyakov, Hsin-Ying Lee, and Chaoyang Wang. DELTA: Dense efficient long-range 3d tracking for any video. In International Conference on Learning Representations (ICLR), 2025. doi: 10.48550/arXiv.2410. 24211

    2025 doi

  56. [65]

    PointWorld: Scaling 3d world models for in-the-wild robotic manipulation

    Wenlong Huang, Yu-Wei Chao, Arsalan Mousavian, Ming-Yu Liu, Dieter Fox, Kaichun Mo, and Fei-Fei Li. PointWorld: Scaling 3d world models for in-the-wild robotic manipulation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20765...

    2026

  57. [66]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. InProceedings of the IEEE/CVF International Conference on Computer Vision (...

    2023

  58. [67]

    doi: 10.48550/arXiv.2512.13644

    doi

  59. [68]

    Rotary position embedding for vision transformer

    Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. InComputer Vision – ECCV 2024, pages 289–305. Springer, 2024

    2024

  60. [69]

    RoFormer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024. doi: 10.1016/j.neucom.2023.127063

    2024

  61. [70]

    Forecasting motion in the wild.arXiv preprint arXiv:2604.01015, 2026

    Neerja Thakkar, Shiry Ginosar, Jacob Walker, Jitendra Malik, Joao Carreira, and Carl Doersch. Forecasting motion in the wild.arXiv preprint arXiv:2604.01015, 2026

    2026

  62. [71]

    CoTracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. CoTracker: It is better to track together. InComputer Vision – ECCV 2024, volume 15120 ofLecture Notes in Computer Science, pages 18–35. Springer, 2024. doi: 10.1007/978-3...

    2024 doi

  63. [72]

    Dex4d: Task-agnostic point track policy for sim-to-real dexterous manipulation.arXiv preprint arXiv:2602.15828, 2026

    Yuxuan Kuang, Sungjae Park, Katerina Fragkiadaki, and Shubham Tulsiani. Dex4d: Task-agnostic point track policy for sim-to-real dexterous manipulation.arXiv preprint arXiv:2602.15828, 2026. 18 MotionForesight Brains, Bots, and Behavior Lab A Implementation Details Here we disc...

    2026

  64. [73]

    SpatialTrackerV2: Advancing 3d point tracking with explicit camera motion

    Yuxi Xiao, Jianyuan Wang, Nan Xue, Nikita Karaev, Yuri Makarov, Bingyi Kang, Xing Zhu, Hujun Bao, Yujun Shen, and Xiaowei Zhou. SpatialTrackerV2: Advancing 3d point tracking with explicit camera motion. InProceedings of the IEEE/CVF International Conference on Computer Vision ...

    2025

  65. [75]

    Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang

    Haotong Lin, Sili Chen, Junhao Liew, Donny Y. Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang. Depth anything 3: Recovering the visual space from any views.arXiv preprint arXiv:2511.10647, 2025. doi: 10.48550/arXiv.2511.10647

    2025 doi arXiv

  66. [79]

    Novaflow: Zero-shot manipulation via actionable flow from generated videos.arXiv preprint arXiv:2510.08568, 2025

    Hongyu Li, Lingfeng Sun, Yafei Hu, Duy Ta, Jennifer Barry, George Konidaris, and Jiahui Fu. Novaflow: Zero-shot manipulation via actionable flow from generated videos.arXiv preprint arXiv:2510.08568, 2025

    2025

  67. [81]

    Estimate observed pointmapsP0:K−1 from the RGB context frames

  68. [82]

    Transform the observed pointmaps into the last-observed camera frame

  69. [83]

    Encode observed RGB frames and pointmaps into visual-geometry latents

  70. [84]

    Replace all future RGB and pointmap context latents with learned mask latents

  71. [85]

    Repeat the reference-frame query latent across allTtime steps

  72. [86]

    Run the frozen video DiT with the frozen tracking adapter and the trained future adapter

  73. [87]

    Decode residual-track latents with the frozen track decoder

  74. [88]

    Add the decoded residuals to the reference point positions to obtain future 3D tracks. 20 MotionForesight Brains, Bots, and Behavior Lab A.6 Evaluation metrics For held-out validation clips with pseudo-ground-truth tracks, we evaluate future average displace- ment error and fi...

  75. [2017]

    doi: 10.1109/ICRA.2017.7989324

    2017

  76. [2022]

    URLhttps://proceedings.mlr.press/v164/mersch22a.html

  77. [2024]

    doi: 10.15607/RSS.2024.XX.092

    2024 doi

  78. [2025]

    doi: 10.48550/arXiv.2503.20314

    doi arXiv

  79. [2026]

    12 MotionForesight Brains, Bots, and Behavior Lab

    doi: 10.48550/arXiv.2601.05237. 12 MotionForesight Brains, Bots, and Behavior Lab

    doi

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.