ContactFlow: A video action conditioning that transfers across embodiments

Enrico Pallotta; Hermann Blum; Jan Nogga; J\"urgen Gall; Sami Azirar; Sven Behnke

REVIEW 2 major objections 5 minor 60 references

ContactFlow: A video action conditioning that transfers across embodiments

T0 review · 2 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Video world models can be steered by the trajectory of contact points between actor and object, making the same model work for human hands and robot grippers.

desk verdict Contact Flow is a genuinely useful action representation and the Q1 video-prediction results are credible—but the zero-shot verifier claim rests on a single 8/10 deployment result with no validation that the synthesized contact flow matches real contact. read the letter →

arxiv 2607.26579 v1 pith:YUQO3ZDI submitted 2026-07-29 cs.RO cs.CV

Sami Azirar , Enrico Pallotta , Jan Nogga , Jürgen Gall , Sven Behnke , Hermann Blum This is my paper

classification cs.ROcs.CV

keywords ContactFlowembodiment-agnosticactionrepresentationvideoworldmodelsmanipulationpredictionmodelingrobotverificationhumandemonstrationtransfergenerationconditioning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that what matters for predicting physical manipulation is not the actor's body but the moving locus of contact between actor and object. It introduces Contact Flow, a compact encoding of 3D contact-point positions and motions projected into the image, and uses it to condition a single video world model trained on both human demonstrations and robot episodes. The central claim is that this embodiment-agnostic signal lets one model transfer zero-shot across unseen robots, scenes, and objects, and can serve as a verifier that rejects unsafe or unsuccessful proposed trajectories before execution. A sympathetic reader would care because current action representations tie world models to one robot's joint space or to a visible hand, limiting transfer; Contact Flow aims to isolate the mechanism that actually moves objects.

What carries the argument

Contact Flow: at each time step, a set of points on the object surface in contact with the actor, each carrying 3D position, 3D displacement to the next frame, and a confidence weight; projected into image space as a sparse 7-channel control video that conditions a latent video diffusion world model. The abstraction does the work: by keeping only the contact locus, it makes human demonstrations and robot executions share one conditioning interface, and lets inference-time contact flow be synthesized from gripper geometry and the recovered object model alone.

What would settle it

Run the verifier on a set of proposed trajectories where object pose is deliberately perturbed by increasing amounts (e.g., 1–10 cm) before synthesizing contact flow, and measure how often the world model's forecast of success flips relative to real execution; if forecasts degrade sharply at small pose errors, the zero-shot verifier claim is false in the regime where it is needed.

Watch

Extended reading notes

Core claim

Central claim: the 3D contact-point trajectory between actor and object is a sufficient conditioning signal for video prediction of manipulation. Each point stores position, per-frame displacement, and confidence; projected to image space they form a sparse 7-channel control video that steers a latent video diffusion model. Dropping hand shape and robot kinematics lets one model train on mixed human and robot data and lets inference synthesize the signal from a planned gripper trajectory plus the recovered object model. The paper reports plausible rollouts on unseen benchmarks and correct real-robot forecasts in 8 of 10 unseen tabletop tasks, where the policy alone completed none.

Load-bearing premise

The synthesized contact flow computed from the planned gripper trajectory and the recovered object model is faithful enough to the contact a real robot will make; if pose estimation is off or execution deviates, the imagined video can diverge from real physics, and the verifier's judgment goes with it.

Editorial extensions

If this is right

A single world model can be trained on mixed human and robot interaction videos, and the same conditioning signal transfers across embodiments with no per-robot adaptation.
Contact Flow can be generated at inference time from the planned end-effector trajectory and the object model, enabling zero-shot verification of proposed trajectories in unseen scenes.
Conditioning on the contact locus rather than whole-actor geometry improves prediction accuracy on held-out robot data and across unseen human- and robot-manipulation benchmarks.
The propose-imagine-verify-act pipeline enables successful open-loop execution on a real fixed-arm robot in unseen tabletop tasks, where the underlying policy alone completed no runs.
Contact Flow can be injected through two different control mechanisms, indicating that the representation, not the control architecture, drives the result.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If Contact Flow is truly sufficient, whole-body or whole-arm conditioning in video world models may be largely wasted capacity; a natural test would be ablating the confidence channel or replacing contact flow with object-surface flow while keeping everything else fixed.
The contact-flow interface could extend beyond hand and gripper manipulation to tool use or multi-contact skills, since it encodes any moving contact locus; the paper's data already include bimanual and tool interactions, but the authors do not claim tool-transfer results.
The verifier claim rests on synthesizing contact flow from a symbolic twin; a concrete next test is to measure how prediction accuracy degrades as object pose error grows, which the paper only partially addresses with its refinement stages.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Punchline: Contact Flow is a good idea, and the Q1 video-prediction results are credible — but the Q2 zero-shot verifier claim is under-supported, and the paper itself admits the twin is "too crude to certify that motion."

The representation is new in the way that matters: it conditions video generation on the trajectory of 3D contact points between actor and object, not on actor masks, keypoints, whole-robot pointmaps, or whole-object flow. That gives a single world model a shared action interface for human and robot video. The data-processing pipeline for extracting Contact Flow from heterogeneous sources is substantial and well described, including the HACO contact filtering and the geometric gripper contact estimation. Testing both ControlNet and VACE conditioning across several backbone scales is thorough. Q1 results are consistent: Contact Flow beats Kinema4D, CTRL-World, and TesserAct on DreamSim on DROID and across held-out and unseen datasets, and the mixture ablations (DROID-only, human-only, mix) show mixed training helps. Masking the actor in both prediction and ground truth is the right choice.

The soft spot is Q2. The evidence is 8/10 correct VLM verdicts on one deployment scenario — a single binomial point with a wide confidence interval, no error bars, no end-to-end success rate, and no comparison against a non-generative verifier. More importantly, the verifier's input flow is synthesized from gripper geometry in the symbolic twin; the paper never checks that this synthesized flow matches what the extraction pipeline would produce from the actual rollout. The pose-refinement pipeline admits a pre-refinement error of several centimetres, and the text says the twin is "too crude to certify that motion." That is a self-admitted limitation on the central claim. There is also a duplicated Q2 paragraph and a placeholder acknowledgments section — easy fixes, but signs the manuscript is not camera-ready.

Who benefits: researchers working on world models for manipulation, cross-embodiment learning, and video conditioning. They will get a useful representation and solid baseline comparisons. The Q1 contribution deserves a serious referee; the Q2 claim needs stronger validation before it is taken as evidence of zero-shot verification.

Recommendation: send to peer review — but the authors should be pushed to release code/data, add sensitivity analysis on the contact-flow extraction thresholds, and most importantly compare synthesized Contact Flow against the flow extracted from real execution footage, plus report end-to-end success rates with more trials. If they close that gap, this becomes a strong paper.

Referee Report

2 major / 5 minor

Summary. The paper introduces Contact Flow, a 7-channel action representation encoding the 3D trajectory of contact points between an actor and an object, projected into image space and used to condition a video diffusion world model. The authors train this world model on a mixture of human hand-object videos and robot demonstrations, evaluate it on held-out DROID clips and several cross-dataset benchmarks, and deploy it in a propose-imagine-verify-act pipeline on a real Franka Panda. The two stated questions are (Q1) whether Contact Flow is an effective conditioning signal for robot manipulation prediction, and (Q2) whether the world model can act as a zero-shot verifier of proposed trajectories in unseen scenes. The paper reports consistent DreamSim improvements over baselines for Q1, and an 8/10 VLM agreement with real-robot outcomes for Q2.

Significance. If the results are robust, Contact Flow is a valuable step toward embodiment-agnostic action conditioning for video world models: it is a compact, clearly defined representation that can be extracted from both human and robot data, and the paper evaluates it across two control-injection mechanisms (ControlNet and VACE), multiple backbone scales, and a real-robot deployment. The experimental design is transparent about masking, held-out splits, and dataset provenance. However, the zero-shot verifier claim currently rests on a small, unperturbed real-world experiment with no sensitivity analysis, so the significance of the Q2 result is not yet established. The Q1 results are informative but would be strengthened by uncertainty quantification.

major comments (2)

[§4 (Q2), §3.3.3] The central Q2 claim that the pipeline is a zero-shot verifier depends on the Contact Flow synthesized from gripper geometry in the symbolic twin being faithful to the contact actually made during open-loop execution. The paper itself states that SAM 3D-Objects leaves 'a pose error of several centimetres' before refinement (§3.3.3) and that the twin is 'too crude to certify that motion' (§4). Yet no sensitivity analysis relates verification accuracy to object-pose error or trajectory-execution error, and there is no comparison between the synthesized Contact Flow and the Contact Flow extracted from a real rollout via the §3.3.2 pipeline. Moreover, the evidence is 8/10 agreements; a binomial 95% CI spans roughly 44–97%, so the point estimate is weak. Without perturbing the input pose/trajectory and measuring downstream VLM agreement, the verifier may be predicting its own idealized input
[§4, Tables 1–2] The headline Q1 comparisons are single point estimates computed over 25 held-out clips, with no confidence intervals, significance tests, or statement about diffusion sampling seeds. Several margins over the Kinema4D baseline are small (e.g., DROID DreamSim 0.035 vs 0.043 in Table 1), and in Table 2 the 14B (mix) model does not uniformly beat the 5B model (e.g., GenieSimOOD DreamSim 0.063 vs 0.050). Because DreamSim is the primary metric and the paper claims 'consistent improvements,' the authors should report variance across repeated generations or clip-level bootstrap confidence intervals, and ideally paired tests, to establish that the observed differences are not sampling noise.

minor comments (5)

[§4 (Q2)] The results paragraph under Q2 is duplicated verbatim; one copy should be removed.
[§4 (Q2)] Typo: 'dooes not enable' should be 'does not enable'.
[Acknowledgments] The manuscript contains placeholder text ('If a paper is accepted...') that should be replaced with actual acknowledgments or removed before submission.
[Table 2] Some rows are typeset ambiguously: labels such as '5B' appear concatenated with metric values (e.g., '0.125B 0.044'), making the table hard to read. Please fix the formatting.
[§3.3.1–3.3.3] Several preprocessing thresholds and mixture proportions are introduced without sensitivity analysis (δ_contact_dist, Hough smoothing window, render-space gates, human/robot data mix). A brief ablation or discussion of their influence would help separate the representation's value from tuned preprocessing.

Circularity Check

0 steps flagged · score 2.0 of 10

Central derivation is self-contained; only a minor, non-load-bearing self-citation prevents a 0.

full rationale

Contact Flow is defined as the trajectory of 3D contact points (Eq. 1) and explicitly excludes post-contact object motion ('it does not include the object motion that happens afterward'), so conditioning on C_{1:T} in Eq. (2) does not make the predicted frames equal to the input by construction; the object dynamics are learned via the flow-matching objective (Eq. 3) from data. The Q1 evaluations condition on ground-truth trajectories and are scored against real recordings, and the cross-dataset tests use held-out or unseen datasets, so no fitted parameter is renamed as a prediction. The Q2 verifier does rely on an idealized inference-time contact flow synthesized from the symbolic twin, and the paper itself explicitly flags this limitation: 'The minimal twin lets the policy propose a motion, but it is too crude to certify that motion.' That is a correctness/robustness concern, not a circular reduction of the derivation to its input. A duplicated deployment paragraph appears twice verbatim; this is a manuscript defect, not circularity. The only self-citation is EgoControl [15] (overlapping authors), used as an example of embodiment-specific joint-based conditioning ('Joint-based representations [10, 15] are expressive but inherently embodiment-specific'); it is not load-bearing for any Contact Flow claim. No uniqueness theorem is imported, no ansatz is smuggled via citation, and no known result is merely renamed. Hence no significant circularity; the minor non-load-bearing self-citation accounts for the score of 2.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The paper's central claim rests on a small number of domain assumptions about contact-centric physics and the faithfulness of the synthesized inference-time contact flow, plus several hand-chosen thresholds in the data-processing pipeline. No physics constants are fitted, and no hidden entities beyond the representation itself are introduced.

free parameters (4)

contact distance threshold δ_contact_dist
Used to binarize hand-object contact for HACO filtering in Sec. 3.3.1; hand-chosen, not reported numerically.
Hough smoothing window (3 frames) for point correspondences = 3 frames
Sec. 3.3.2 says 'empirically found to be the best operating point'; no sensitivity analysis.
Render-space gate thresholds (mask IoU, depth inlier fraction, RGB cosine similarity ≥0.95) = ≥0.95 for RGB cosine similarity
Hand-set acceptance criteria in the object-pose refinement of Sec. 3.3.3; affect the quality of the synthesized contact flow at inference.
Data mixture proportions (human vs robot training data)
Not reported; the cross-embodiment transfer claim depends on how human and robot data are balanced in the mix.

assumptions (5)

domain assumption Physical manipulation is governed by contact; an object's movement is only determined by where and how force is applied to it.
Foundational motivation in the Introduction (paragraph 2); a reasonable but unproven simplification of mechanics.
domain assumption The same contact motion should produce the same representation, independent of whether a human hand or a robot gripper created it.
Sec. 3.1; the cross-embodiment transfer relies on this equivalence between contact-flow signals from different actors.
domain assumption Contact flow excludes passive dynamics; the model must learn passive object motion from data.
Sec. 3.1; the representation intentionally omits post-contact outcomes, assuming the generative model can infer them from the contact signal.
domain assumption A calibrated exocentric camera with reliable metric depth is available for training and inference.
Stated in Sec. 3.3 and Limitations; the entire contact-flow extraction depends on this.
domain assumption The VLM judge reliably determines task success from generated video.
Q2 relies on VLM verdicts matching real outcomes; only 8/10 agreement is evaluated, so this is a load-bearing assumption.

invented entities (1)

Contact Flow
purpose: Embodiment-agnostic action conditioning signal for video world models.
A new representation, not a physical entity; its utility is demonstrated only in-paper, with no external falsifiable prediction outside the paper's own experiments.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ContactFlow: A video action conditioning that transfers across embodiments." pith.science (2026). https://pith.science/paper/YUQO3ZDI

@misc{pith2026260726579,
  author       = {Pith},
  title        = {Pith review of: ContactFlow: A video action conditioning that transfers across embodiments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YUQO3ZDI}},
  note         = {Machine review of arXiv:2607.26579}
}

read the original abstract

World models offer a promising route toward robot planning by enabling agents to imagine and verify the consequences of actions before execution. However, current video-based world models often struggle to capture the physical constraints that govern manipulation, particularly contact. Further, their action conditioning is often constrained to specific embodiments such as parallel grippers. We propose \emph{Contact Flow}, an embodiment-agnostic action representation that encodes manipulation through the trajectory of 3D contact points between an actor and a target object. By discarding actor-specific appearance and kinematics, Contact Flow provides a shared conditioning signal for both human demonstrations and robotic execution. Therefore, we can train a large-scale video generative model on both human and robotic object interaction videos conditioned on Contact Flow, yielding a world model that predicts physically plausible manipulation outcomes. We integrate this model into a propose-imagine-verify-act pipeline, where generated rollouts are assessed by a vision-language model before execution. Experiments on the DROID dataset and real-world tabletop manipulation tasks demonstrate that Contact Flow enables transfer between human demonstrations and different robotic embodiments.

Figures

Figures reproduced from arXiv: 2607.26579 by the authors.

**Figure 1.** Closed-loop deployment with Contact Flow. On the left, a single RGBD view instantiates a symbolic twin simulator in which a π0.5 policy proposes an end-effector trajectory. On the right, our world model imagines the rollout conditioned on the initial frame and Contact Flow from that trajectory, showing the real rollout above and the simulated rollout below. 4.1 Experimental Setup Training data. We train on DROID, [… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 10 linked inside Pith

[1]

Y . Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. InAdvances in Neural Informa- tion Processing Systems, volume 36, 2023

2023
[2]

Y . Du, S. Yang, P. Florence, F. Xia, A. Wahid, P. Sermanet, T. Yu, P. Abbeel, J. B. Tenen- baum, L. Kaelbling, et al. Video language planning. InInternational Conference on Learning Representations, volume 2024, pages 31138–31155, 2024

2024
[3]

H. Wu, Y . Jing, C. Cheang, G. Chen, J. Xu, X. Li, M. Liu, H. Li, and T. Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. InInternational Con- ference on Learning Representations, 2024. URLhttps://openreview.net/forum?id= NxoFmGgWC9

2024
[4]

Liang, P

J. Liang, P. Tokmakov, R. Liu, S. Sudhakar, P. Shah, R. Ambrus, and C. V ondrick. Video generators are robot policies.arXiv preprint arXiv:2508.00795, 2025. doi:10.48550/arXiv. 2508.00795

arXiv doi:10.48550/arxiv.2508.00795 2025
[5]

S. Zhou, Y . Du, J. Chen, Y . Li, D.-Y . Yeung, and C. Gan. RoboDreamer: Learning composi- tional world models for robot imagination. InProceedings of the 41st International Confer- ence on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 61885–61896. PMLR, 2024. URLhttps://proceedings.mlr.press/v235/zhou24f. html

2024
[6]

J. Jang, S. Ye, Z. Lin, J. Xiang, J. Bjorck, Y . Fang, F. Hu, S. Huang, K. Kundalia, Y .-C. Lin, L. Magne, A. Mandlekar, A. Narayan, Y . L. Tan, G. Wang, J. Wang, Q. Wang, Y . Xu, X. Zeng, K. Zheng, R. Zheng, M.-Y . Liu, L. Zettlemoyer, D. Fox, J. Kautz, S. Reed, Y . Zhu, and L. Fan. Dreamgen: Unlocking generalization in robot learning through video world...

2025
[7]

Liang, R

J. Liang, R. Liu, E. Ozguroglu, S. Sudhakar, A. Dave, P. Tokmakov, S. Song, and C. V ondrick. Dreamitate: Real-world visuomotor policy learning via video generation. InProceedings of The 8th Conference on Robot Learning, volume 270 ofProceedings of Machine Learning Re- search, pages 3943–3960. PMLR, 2025. URLhttps://proceedings.mlr.press/v270/ liang25b.html

2025
[8]

F. Zhu, H. Wu, S. Guo, Y . Liu, C. Cheang, and T. Kong. IRASim: A fine-grained world model for robot manipulation. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9834–9844, October 2025

2025

Show all 60 references

[9]

Y . Wang, R. Syed, F. Wu, M. Zhang, A. Onol, J. Barreiros, H. Nayyeri, T. Dear, H. Zhang, and Y . Li. Interactive world simulator for robot policy training and evaluation. InRobotics: Science and Systems, 2026. URLhttps://www.yixuanwang.me/interactive_world_sim/

2026
[10]

Agarwal, A

N. Agarwal, A. Ali, M. Bala, Y . Balaji, E. Barker, T. Cai, P. Chattopadhyay, Y . Chen, Y . Cui, Y . Ding, et al. Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575, 2025. 9

2025 arXiv
[11]

A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, J. Zeng, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

2025 arXiv
[12]

Jiang, Z

Z. Jiang, Z. Han, C. Mao, J. Zhang, Y . Pan, and Y . Liu. Vace: All-in-one video creation and editing. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 17191–17202, 2025

2025
[13]

Bansal, Z

H. Bansal, Z. Lin, T. Xie, Z. Zong, M. Yarom, Y . Bitton, C. Jiang, Y . Sun, K.-W. Chang, and A. Grover. Videophy: Evaluating physical commonsense for video generation. InInternational Conference on Learning Representations, volume 2025, pages 102075–102121, 2025

2025
[14]

Motamed, L

S. Motamed, L. Culp, K. Swersky, P. Jaini, and R. Geirhos. Do generative video models under- stand physical principles? InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 948–958, 2026

2026
[15]

Pallotta, S

E. Pallotta, S. M. Azar, L. Doorenbos, S. Ozsoy, U. Iqbal, and J. Gall. Egocontrol: Controllable egocentric video generation via 3d full-body poses. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 4269–4279, June 2026

2026
[16]

G. Li, B. Zhao, J. Yang, and L. Sevilla-Lara. Mask2iv: Interaction-centric video generation via mask trajectories. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 6091–6099, 2026

2026
[17]

H. Yan, H. Yu, Z. Zhong, W. Yuan, X. Gong, Z. Luo, C. Heyu, J. Li, W. Song, S. Zhou, et al. Open-world hand-object interaction video generation based on structure and contact-aware representation.arXiv preprint arXiv:2512.01677, 2025

2025
[18]

H. Zhao, X. Liu, M. Xu, Y . Hao, W. Chen, and X. Han. Taste-rob: Advancing video generation of task-oriented hand-object interaction for generalizable robotic manipulation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 27683–27693, 2025

2025
[19]

Khazatsky, K

A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. InRobotics: Science and Systems, 2024

2024
[20]

Zhang, A

L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models. InProceedings of the IEEE/CVF international conference on computer vision, pages 3836–3847, 2023

2023
[21]

Y . Bai, L. Yang, G. Eskandar, F. Shen, M. Altillawi, Z. Liu, and G. Kutyniok. DRAW2ACT: Turning depth-encoded trajectories into robotic demonstration videos.arXiv preprint arXiv:2512.14217, 2025

2025
[22]

W. Lin, J. Zhang, K. Cai, and K. Wang. Storm: Search-guided generative world models for robotic manipulation.arXiv preprint arXiv:2512.18477, 2025

2025
[23]

H. Zhen, Q. Sun, H. Zhang, J. Li, S. Zhou, Y . Du, and C. Gan. Tesseract: learning 4d embodied world models.arXiv preprint arXiv:2504.20995, 2025

2025 arXiv
[24]

Akkerman, H

R. Akkerman, H. Feng, M. J. Black, D. Tzionas, and V . F. Abrevaya. Interdyn: Controllable interactive dynamics with video diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12467–12479, 2025

2025
[25]

Y . Wang, C. Wen, H. Guo, S. Peng, M. Qin, H. Bao, X. Zhou, and R. Hu. Precise action-to- video generation through visual action prompts. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 12713–12724, 2025. 10

2025
[26]

W. Jin, Q. Dai, C. Luo, S.-H. Baek, and S. Cho. Flovd: Optical flow meets video diffusion model for enhanced camera-controlled video synthesis. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 2040–2049, 2025
[27]

Koroglu, H

M. Koroglu, H. Caselles-Dupr ´e, G. Jeanneret, and M. Cord. Onlyflow: Optical flow based motion conditioning for video diffusion models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 6226–6236, 2025

2025
[28]

D. Geng, C. Herrmann, J. Hur, F. Cole, S. Zhang, T. Pfaff, T. Lopez-Guevara, Y . Aytar, M. Ru- binstein, C. Sun, O. Wang, A. Owens, and D. Sun. Motion prompting: Controlling video generation with motion trajectories. InProceedings of the IEEE/CVF Conference on Com- puter Visio...

2025
[29]

Brahmbhatt, C

S. Brahmbhatt, C. Ham, C. C. Kemp, and J. Hays. Contactdb: Analyzing and predicting grasp contact via thermal imaging. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8709–8719, 2019

2019
[30]

Brahmbhatt, C

S. Brahmbhatt, C. Tang, C. D. Twigg, C. C. Kemp, and J. Hays. Contactpose: A dataset of grasps with object contact and hand pose. InComputer Vision – ECCV 2020, pages 361–378. Springer, 2020

2020
[31]

Grady, C

P. Grady, C. Tang, C. D. Twigg, M. V o, S. Brahmbhatt, and C. C. Kemp. Contactopt: Opti- mizing contact to improve grasps. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1471–1481, 2021

2021
[32]

T. Kwon, B. Tekin, J. St ¨uhmer, F. Bogo, and M. Pollefeys. H2O: Two hands manipulating objects for first person interaction recognition. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10138–10148, 2021

2021
[33]

Sundermeyer, A

M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-graspnet: Efficient 6-DoF grasp generation in cluttered scenes. InIEEE International Conference on Robotics and Au- tomation (ICRA), pages 13438–13444, 2021

2021
[34]

T. Pang, H. J. T. Suh, L. Yang, and R. Tedrake. Global planning for contact-rich manipulation via local smoothing of quasi-dynamic contact models.IEEE Transactions on Robotics, 39(6): 4691–4711, 2023

2023
[35]

M. Xu, T. Zhang, T. Liu, Z. Chen, X. Han, and Z. Liu. Kinema4d: Kinematic 4d world modeling for spatiotemporal embodied simulation.arXiv preprint arXiv:2603.16669, 2026

2026
[36]

Y . Chen, P. Li, J. Yang, K. He, X. Wu, Y . Xu, K. Wang, J. Liu, N. Liu, Y . Huang, et al. Bridgev2w: Bridging video generation models to embodied world models via embodiment masks.arXiv preprint arXiv:2602.03793, 2026

2026
[37]

M. Xu, Z. Xu, Y . Xu, C. Chi, G. Wetzstein, M. Veloso, and S. Song. Flow as the cross-domain manipulation interface. In8th Annual Conference on Robot Learning (CoRL), 2024

2024
[38]

C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y . Gao, and P. Abbeel. Any-point trajectory modeling for policy learning. InRobotics: Science and Systems (RSS), 2024

2024
[39]

H. Zhi, P. Chen, S. Zhou, Y . Dong, Q. Wu, L. Han, and M. Tan. 3DFlowAction: Learning cross-embodiment manipulation from 3d flow world model.arXiv preprint arXiv:2506.06199, 2025

2025 arXiv
[40]

Z.-H. Yin, S. Yang, and P. Abbeel. Object-centric 3d motion field for robot learning from human videos.Advances in Neural Information Processing Systems, 38:55923–55943, 2026

2026
[41]

B. Wen, M. Trepte, J. Aribido, J. Kautz, O. Gallo, and S. Birchfield. FoundationStereo: Zero- shot stereo matching. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5249–5260, 2025. 11

2025
[42]

Keetha, N

N. Keetha, N. M ¨uller, J. Sch¨onberger, L. Porzi, Y . Zhang, T. Fischer, A. Knapitsch, D. Zauss, E. Weber, J. Luiten, M. Lopez-Antequera, S. Rota Bul`o, C. Richardt, D. Ramanan, S. Scherer, and P. Kontschieder. MapAnything: Universal feed-forward metric 3D reconstruction. In ...

2026
[43]

Pavlakos, D

G. Pavlakos, D. Shan, I. Radosavovic, A. Kanazawa, D. Fouhey, and J. Malik. Reconstructing hands in 3d with transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9826–9836, 2024

2024
[44]

Romero, D

J. Romero, D. Tzionas, and M. J. Black. Embodied hands: Modeling and capturing hands and bodies together.ACM Transactions on Graphics (Proc. SIGGRAPH Asia), 36(6):245:1– 245:17, 2017

2017
[45]

Carion, L

N. Carion, L. Gustafson, Y .-T. Hu, S. Debnath, R. Hu, D. Suris, et al. SAM 3: Segment anything with concepts, 2025

2025
[46]

Honnibal, I

M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd. spaCy: Industrial-strength natural language processing in python, 2020

2020
[47]

G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

2023 arXiv
[48]

D. S. Jung and K. M. Lee. Learning dense hand contact estimation from imbalanced data. Advances in Neural Information Processing Systems, 2025

2025
[49]

Y . Liu, H. Yang, X. Si, L. Liu, Z. Li, Y . Zhang, Y . Liu, and L. Yi. TACO: Benchmarking generalizable bimanual tool-ACtion-object understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21740–21751, 2024

2024
[50]

X. Zhan, L. Yang, Y . Zhao, K. Mao, H. Xu, Z. Lin, K. Li, and C. Lu. OAKINK2: A dataset of bimanual hands-object manipulation in complex task completion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 445–456, 2024

2024
[51]

S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. InComputer Vision – ECCV 2024, pages 38–55. Springer, 2024

2024
[52]

Huang, J

W. Huang, J. Zhang, S. Li, T. Jia, J. Duan, Y . Cheng, J. Cho, M. Wallingford, R. Soraki, C. D. Kim, S. Liu, D. Clay, T. Anderson, W. Han, A. Farhadi, B. Hariharan, Z. Ren, and R. Krishna. WildDet3D: Scaling promptable 3D detection in the wild, 2026

2026
[53]

C. Yuan, S. Joshi, S. Zhu, H. Su, H. Zhao, and Y . Gao. RoboEngine: Plug-and-play robot data augmentation with semantic robot segmentation and background generation. In2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7622– 7629, 2025

2025
[54]

H. Li, Z. Wang, Z.-h. Ding, S. Yang, Y . Chen, Y . Tian, X. Hu, T. Wang, D. Lin, F. Zhao, et al. Robointer: A holistic intermediate representation suite towards robotic manipulation. InThe Fourteenth International Conference on Learning Representations, 2026

2026
[55]

Chen, F.-J

X. Chen, F.-J. Chu, P. Gleize, K. J. Liang, A. Sax, H. Tang, W. Wang, M. Guo, T. Hardin, X. Li, et al. Sam 3d: 3dfy anything in images.arXiv preprint arXiv:2511.16624, 2025

2025 arXiv
[56]

V . Ye, R. Li, J. Kerr, M. Turkulainen, B. Yi, Z. Pan, O. Seiskari, J. Ye, J. Hu, M. Tancik, and A. Kanazawa. gsplat: An open-source library for Gaussian splatting.Journal of Machine Learning Research, 2024. 12

2024
[57]

B. Liu, Y . Zhu, C. Gao, Y . Feng, Q. Liu, Y . Zhu, and P. Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. InAdvances in Neural Information Processing Systems, volume 36, pages 44776–44791, 2023

2023
[58]

Black, N

Physical Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Es- mail, M. Equi, C. Finn, K. Hausman, B. Ichter, S. Levine, K. Pertsch, Q. Vuong, H. Walke, et al.π 0.5: a vision-language-action model with open-world generalization.arXiv preprint arXiv:250...

2025 arXiv
[59]

Y . Guo, L. X. Shi, J. Chen, and C. Finn. Ctrl-world: A controllable generative world model for robot manipulation.arXiv preprint arXiv:2510.10125, 2025

2025 arXiv
[60]

S. Fu, N. Tamir, S. Sundaram, L. Chai, R. Zhang, T. Dekel, and P. Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data.arXiv preprint arXiv:2306.09344, 2023. 13

2023 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

Y . Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. InAdvances in Neural Informa- tion Processing Systems, volume 36, 2023

2023

[2] [2]

Y . Du, S. Yang, P. Florence, F. Xia, A. Wahid, P. Sermanet, T. Yu, P. Abbeel, J. B. Tenen- baum, L. Kaelbling, et al. Video language planning. InInternational Conference on Learning Representations, volume 2024, pages 31138–31155, 2024

2024

[3] [3]

H. Wu, Y . Jing, C. Cheang, G. Chen, J. Xu, X. Li, M. Liu, H. Li, and T. Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. InInternational Con- ference on Learning Representations, 2024. URLhttps://openreview.net/forum?id= NxoFmGgWC9

2024

[4] [4]

Liang, P

J. Liang, P. Tokmakov, R. Liu, S. Sudhakar, P. Shah, R. Ambrus, and C. V ondrick. Video generators are robot policies.arXiv preprint arXiv:2508.00795, 2025. doi:10.48550/arXiv. 2508.00795

arXiv doi:10.48550/arxiv.2508.00795 2025

[5] [5]

S. Zhou, Y . Du, J. Chen, Y . Li, D.-Y . Yeung, and C. Gan. RoboDreamer: Learning composi- tional world models for robot imagination. InProceedings of the 41st International Confer- ence on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 61885–61896. PMLR, 2024. URLhttps://proceedings.mlr.press/v235/zhou24f. html

2024

[6] [6]

J. Jang, S. Ye, Z. Lin, J. Xiang, J. Bjorck, Y . Fang, F. Hu, S. Huang, K. Kundalia, Y .-C. Lin, L. Magne, A. Mandlekar, A. Narayan, Y . L. Tan, G. Wang, J. Wang, Q. Wang, Y . Xu, X. Zeng, K. Zheng, R. Zheng, M.-Y . Liu, L. Zettlemoyer, D. Fox, J. Kautz, S. Reed, Y . Zhu, and L. Fan. Dreamgen: Unlocking generalization in robot learning through video world...

2025

[7] [7]

Liang, R

J. Liang, R. Liu, E. Ozguroglu, S. Sudhakar, A. Dave, P. Tokmakov, S. Song, and C. V ondrick. Dreamitate: Real-world visuomotor policy learning via video generation. InProceedings of The 8th Conference on Robot Learning, volume 270 ofProceedings of Machine Learning Re- search, pages 3943–3960. PMLR, 2025. URLhttps://proceedings.mlr.press/v270/ liang25b.html

2025

[8] [8]

F. Zhu, H. Wu, S. Guo, Y . Liu, C. Cheang, and T. Kong. IRASim: A fine-grained world model for robot manipulation. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9834–9844, October 2025

2025