Robotics · arXiv:2608.18077 v1 · submitted 18 Aug 2026 · 67 references
Hydra-0: Action Flow for Generalist World Modeling and Control
The paper says
“We introduce Hydra-0, a generalist world model conditioned on action flow, which represents robot actions as pixel motion. This shared visual interface enables generalist world modeling and control by learning action consequences across embodiments, tasks, environments, and video-generation backbones.”
more of the abstract
Our best configuration achieves 90.4% lower robot-motion error and 60.2% lower object-motion error than our action-conditioned baseline, while supporting zero-shot composition and data-efficient adaptation. On the RoboLab benchmark, Hydra-0 achieves a Pearson correlation of r=0.96 between replayed and reference success rates. Finally, we uncover an emergent inverse mode of this interface: a world action model that predicts compatible robot motion from desired object flow transferred from a human demonstration. A trained action head maps the resulting latent features to executable actions without requiring task-specific expert robot demonstrations. Together, these results demonstrate the potential of action flow as a shared control interface connecting heterogeneous training data, open-loop policy evaluation, and robot control.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 27 Aug 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A strong forward-mode world modeling paper on action flow; the inverse-control and train/deploy flow-consistency claims need quantitative support. This line is wrong
3 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 67 references say.
The paper says, at §2.1, §8.2, §4.5
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The paper's central premise is that the same camera-plane action flow is used for training and deployment, but the two construction routes are statistically different signals. Training-time flow comes from AllTracker dense tracks on a 128x128 query grid, classified by grounded masks into embodiment/object pools (Section 2.1, video-only construction), while deployment flow is produced by propagating robot-surface points through Isaac Lab link transforms and projecting them with Equation (1), with depth-buffer visibility (Section 2.1, geometry-aware construction; Section 8.2).
more of objection 1
AllTracker tracks carry tracker noise, drift, and mask-classification errors, whereas projected flow is smooth rigid-body kinematics with explicit occlusions. The manuscript calls both 'the same trajectory representation,' but no experiment measures the resulting distribution shift, and no training-mixture proportion by construction route is reported. Table 2 and Section 4.5 evaluate with geometry-derived flow, so the model may be conditioned at deployment on flow statistics it never learned; this concern is independent of simulator-to-real fidelity. I request a diagnostic experiment that trains with one route and evaluates with the other in both directions, a report of the training mixture, and, if the gap is large, noise augmentation or calibration jitter to close it.
The paper says, at §4.2, Table 2, Abstract
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The headline error reductions (90.4% robot-motion, 60.2% object-motion) compare the best Wan2.2 A14B configuration against a fine-tuned Cosmos 2.5 baseline, confounding backbone, model size, and sampling procedure with the action representation; the best row is also a 4-step distilled student, adding an inference-cost difference.
more of objection 2
The within-Cosmos comparison in Table 2 is cleaner, but the text's statement that it 'isolates the conditioning representation' (Section 4.2) is still too strong: the flow-conditioned Cosmos model adds a 17-channel motion side input and a zero-initialized patch projection, trains LoRA adapters, and uses a different conditioning signal, whereas the baseline uses native relative 6D actions and no motion input. Please provide a matched ablation in which the only intentional change is the action representation, and report training budgets and total parameter counts; the abstract-level claims should be based on that comparison rather than the cross-backbone best.
The paper says, at §4.5, §6
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The open-loop policy-evaluation claim should be stated with its limitations. In Section 4.5 the policy is never queried during generation; Hydra-0 is conditioned on the recorded achieved end-effector trajectory, so the evaluation measures how faithfully the world model reproduces observed rollouts, not how well it evaluates a policy's output commands.
more of objection 3
Success/failure judgments come from a single human rater with no reported inter-rater agreement, and the paper's own Section 6 acknowledges that only open-loop evaluation is performed. I recommend adding inter-rater statistics, reporting correlation at a finer granularity than the policy-task aggregate if possible, and clearly distinguishing 'outcome preservation under trajectory replay' from full open-loop policy evaluation.
6 minor objections
The paper says, at §4.3, Figure 7
Pith says: minor objection 1 of 6
The text lists training-data fractions of 1%, 2%, 4%, 20%, 40%, 60%, 80%, and 100%, but the horizontal axis in Figure 7 shows only 0, 1, 4, 20, and 100; please clarify the axis scaling and whether intermediate fractions are omitted.
The paper says, at §2.2, Eq. (5)
Pith says: minor objection 2 of 6
The expectation in Equation (5) is written over o0:H, F, t, and epsilon, but the conditioning variables c and C_motion are not listed in the subscript; please make the notation explicit.
show 4 more
The paper says, at Table 2
Pith says: minor objection 3 of 6
FID and FVD are reported without confidence intervals or significance tests, and many 'best' cells differ by less than one point; please add intervals or error bars, or explicitly note which differences are not statistically evaluated.
The paper says, at §8.1
Pith says: minor objection 4 of 6
The appendix refers to 'fig. 11' with inconsistent capitalization; please use 'Figure 11' throughout.
The paper says, at §4.6, §6
Pith says: minor objection 5 of 6
The inverse world-action-model result is described as qualitative and Section 6 admits centimeter-scale grasp imprecision; please state the number of trials performed for the flexible-pipe-bending demonstration and report at least a simple success count so the proof-of-concept is reproducible.
The paper says, at Overall
Pith says: minor objection 6 of 6
The paper gives a project page but does not state whether code, checkpoints, or exact action-flow construction scripts will be released; please add an explicit availability statement.
What would settle it
Use the described data pipeline to construct the identical held-out clips under two conditioning-flow routes: (A) AllTracker dense tracks classified by embodiment/object masks (training route) and (B) geometry-aware projection of recorded robot states through Eq. (1) with depth-buffer visibility (deployment route). Feed both into the same trained checkpoint and compare gripper/object EPE, FVD, and the Section 4.5 replay-correlation on the same clips. To make the test conclusive, train two variants, one trained only on route A and one only on route B, then cross-evaluate each on the other route. If EPE or FVD shifts by more than roughly 10% relative under cross-evaluation, the 'same representation' premise is violated and deployment conditions the model on a distribution it has not learned to interpret.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 27 Aug 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Here's my read on Hydra-0. The core idea is genuinely useful: replace native joint/end-effector action conditioning with camera-plane point trajectories ('action flow'), computed from executable commands via physics rollouts and robot geometry. The paper shows this interface portably transfers across three video backbones and five datasets, with consistently lower motion error than a fine-tuned Cosmos 2.5 baseline. That forward-mode result looks solid, and the multi-embodiment training corpus (1.5M windows) is a real contribution. The open-loop policy evaluation (r=0.96 across RoboLab) is a believable, useful sanity check. Credit where due: the authors are careful to cite ATI and Wan-Move as the trajectory-conditioning precedents, and they build on those cleanly.
The soft spots are real but not fatal. The biggest is the one the stress-test flagged: training-time flow is mostly AllTracker dense tracks (noisy, drifting, from video-only datasets), while deployment flow is projected rigid-body kinematics from Isaac Lab. The paper calls both 'the same trajectory representation,' but that's a data-type identity, not a distributional one. No experiment separates the two routes or reports the training mixture per source. If the model mostly saw tracker-style tracks, deployment-time projected flow could be out-of-distribution, which would undermine the forward-deployment and policy-evaluation claims. This is testable: train on video-only data, evaluate with geometry-derived flow, and vice versa. The paper should do it.
Second, the inverse world action model is only a qualitative single-task demo. The authors themselves call it proof-of-concept, so the abstract's 'emergent inverse mode' oversells it slightly. The zero-shot composition claim is likewise not backed by a dedicated quantitative experiment.
Third, the headline 90.4%/60.2% reductions compare against a single fine-tuned Cosmos 2.5 baseline. The controlled comparison on the same backbone (Ours Cosmos 2.5 vs Cosmos 2.5) is fairer and still shows clear gains, but the headline numbers mix in the Wan2.2 backbone switch. I'd point reviewers to the within-backbone comparison.
The hyperparameters (beta=220, K=2, sampling mode probabilities, filter thresholds) have no ablations. Minor, but relevant for reproducibility. The paper would benefit from releasing code and data artifacts.
Who this is for: researchers working on video world models for robotics, trajectory-conditioned video generation, and policy evaluation. It deserves a serious referee — the forward-mode results are strong enough that the field needs to engage with them. I'd recommend conditional accept for the record, with the train/deploy flow mismatch and inverse-control quantification as the main revision asks.
The strongest objection
The central premise is that the same camera-plane action flow is used for training and deployment. The paper actually constructs two different signals. Section 3 builds the training corpus from dense point tracks recovered by AllTracker on a 128x128 query grid and assigns them to embodiment/object pools with grounded masks (Section 2.1, 'video-only construction'). Deployment (and the main evaluations) instead constructs flow by sampling points on the robot surface, propagating them through link transforms produced by an Isaac Lab controller/physics rollout, and projecting them through camera extrinsics/intrinsics with Eq. (1), with visibility determined by a depth buffer (Section 2.1, 'geometry-aware construction'; Section 8.2). These are not the same signal: AllTracker tracks carry tracker noise, drift, and mask-classification errors, while projected flow is smooth rigid-body kinematics with explicit occlusions. The paper calls both the 'same trajectory representation,' but that is a data-type identity, not a statistical one. No experiment in Section 4 separates this confound: Table 2 and Section 4.5 evaluate with geometry-derived flow (recorded robot states or end-effector trajectories), not with the video-only construction, and the training mixture of the two routes is not reported by source. Consequently, the model may be conditioned, at deployment, on flow statistics it never learned. This concern is independent of the simulator-to-real gap: even a perfect Isaac Lab rollout would fail if the projection route is out-of-distribution for a model trained predominantly on tracker-style tracks. The transfer, data-efficiency, and deployment claims all presuppose that a single flow interface spans both routes, so this mismatch is the most load-bearing unvalidated step.
The reader's weakest assumption points at simulator-to-real fidelity and calibration error. I agree that this is a real gap, but the sharper failure mode is upstream: the paper never establishes that the flow representation used to train the model is the same statistical object as the flow used at deployment. The textual evidence for this is strong: Section 3 describes training tracks from AllTracker on a dense query grid, while Section 2.1 and Eq. (1) describe deployment flow as Isaac Lab link transforms projected through camera calibration. The phrase 'same trajectory representation' holds only at the level of data type. If the two constructions are not interchangeable, the forward-deployment claims fail even with a perfect simulator. This is not an external-consensus disagreement; it is an internally testable question about the method's own interface. The paper does have genuine supporting evidence: a controlled within-backbone comparison (Ours Cosmos 2.5 vs. Cosmos 2.5 native action), consistent gains across three backbones, and a substantive open-loop policy-evaluation correlation. These justify keeping the conditional-acceptance posture rather than escalating to rejection. The inverse world-action model is also only qualitative, but it is a downstream use and the paper labels it a proof of concept; the flow-construction mismatch is more load-bearing because it underlies the forward world-modeling mechanism itself.
The referee's report
Action flow is a plausible and attractive idea: representing actions in image coordinates is embodiment-agnostic and could permit multi-embodiment training of a single world model. The paper's strengths are the scale and diversity of the training corpus (about 1.56M windows across seven sources), the portability of the conditioning interface across Cosmos 2.5 and two Wan2.2 variants, the unusually detailed implementation and evaluation protocol, and the breadth of downstream evaluations, including data-efficiency curves, inference-speed benchmarks, and open-loop policy evaluation. If the central claims held, this would be a meaningful step toward generalist video world models for robotics. However, the mismatch between training-time and deployment-time flow construction, and the confounded headline comparison against a cross-backbone baseline, mean that the strongest quantitative claims are not yet established by the experiments as reported.
Pith's reading of the paper (a machine's summary, not the authors')
The paper argues that a robot's action can be communicated to a video-generation world model not as joint angles or end-effector commands, but as sparse trajectories of visible pixels: action flow, showing where the robot's own surface will move in the camera image. Because this representation lives in image coordinates, the same world model can train on single-arm robots, bimanual arms, handheld grippers, and human hands, and can then be conditioned by any new embodiment whose geometry can be projected into the camera. The paper reports that this conditioning lowers robot-motion error by 90.4% and object-motion error by 60.2% relative to an action-conditioned video baseline, and that open-loop replays of five policies on a simulation benchmark correlate with reference success rates at $r = 0.96$. The same interface is inverted: given desired object motion extracted from a human demonstration, the model predicts compatible robot motion and a small learned head converts its latent features into executable actions. If these results hold, action flow gives robot learning a common visual language that separates what the robot should do from how a particular body can do it.
What carries the argument. The load-bearing object is action flow $\mathcal{F} = \{\tau_n\}_{n=1}^{N}$, a set of sparse camera-plane trajectories $\tau_n = \{(\mathbf{x}_{n,t}, m_{n,t})\}_{t=0}^{H}$ giving the image position and visibility of tracked robot-surface or object points over the prediction horizon. Equation (1) constructs it geometrically by projecting link-frame surface points through camera extrinsics and intrinsics; Equation (3) injects it into the video model by bilinearly sampling first-frame latent features at each trajectory's source and propagating them along its future latent-grid locations with Gaussian weights, forming a motion condition $C_{\mathrm{motion}} = (M, g)$ that is concatenated with the noisy video latent during flow-matching denoising. A separate lightweight action head reads the denoised latent tokens to recover executable actions in the inverse mode. This machinery is what lets one model treat embodiment motion and task intent in the same pixel-aligned format.
Core claim. The central discovery is that kinematically grounded image-plane action flow is a sufficient conditioning interface for a generalist video world model. For deployment, a candidate command is executed through a controller-and-physics rollout, robot-surface points are propagated through link transforms, and camera intrinsics and extrinsics project those points into the observed image plane; moderate calibration error perturbs the conditioning flow without changing the target video, so the model can tolerate imperfect alignment. During training, the same flow representation is recovered from pure video by dense tracking plus embodiment and object segmentation, which lets heterogeneous multi-embodiment data share one predictive model. In the forward direction the model predicts the visual consequence of a candidate command; in the inverse direction, desired object flow alone elicits compatible robot motion, and a lightweight action head decodes that motion into executable commands. The paper thereby claims action flow is not merely a conditioning trick but a shared interface for transfer, open-loop policy evaluation, and control.
The premise it rests on. The whole forward-deployment chain rests on the assumption that executing a command in the physics simulator yields the same visible robot motion as executing it on the real robot, and that camera calibration and robot geometry are accurate enough for the projected pixel flow to line up with the observed video.
If the paper is right (Pith's inferences, not the authors')
A single world model can be trained on interaction videos from many embodiments, because embodied motion is normalized into image-plane flow.
At deployment, any robot whose geometry and camera calibration are known can condition the same model without retraining, so new embodiments inherit dynamics learned from all training data.
Open-loop policy evaluation becomes practical: replaying recorded gripper flow into the model and measuring simulated success ranks policies with high correlation.
Task specification can be separated from execution: object flow extracted from a human demonstration can drive a robot through the inverse action head.
The interface is portable across video-generation backbones, since the motion condition is injected as a visual side channel rather than a backbone-specific command token.
If object-flow conditioning reliably elicits compatible robot motion, action flow could serve as a dense reward or success criterion: compare predicted object flow under a candidate command with desired flow and use the gap for planning or closed-loop correction.
The inverse mode could be trained in a self-supervised loop: roll out the forward model under many commands, and use its predicted object flow as paired supervision for the inverse action head, reducing dependence on real-robot rollouts.
The centimeter-scale grasp imprecision the paper reports may be attackable by conditioning action flow on depth or force alongside pixels, because the current interface is purely geometric and cannot sense contact slip.
Because action flow normalizes away embodiment, the same interface should extend from deformable-object manipulation to rigid assembly or tool use, where contact constraints dominate; the paper's evidence is limited to deformable tasks.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
We introduce Hydra-0, a generalist world model conditioned on action flow, which represents robot actions as pixel motion. This shared visual interface enables generalist world modeling and control by learning action consequences across embodiments, tasks, environments, and video-generation backbones. Our best configuration achieves 90.4% lower robot-motion error and 60.2% lower object-motion error than our action-conditioned baseline, while supporting zero-shot composition and data-efficient adaptation. On the RoboLab benchmark, Hydra-0 achieves a Pearson correlation of r=0.96 between replayed and reference success rates. Finally, we uncover an emergent inverse mode of this interface: a world action model that predicts compatible robot motion from desired object flow transferred from a human demonstration. A trained action head maps the resulting latent features to executable actions without requiring task-specific expert robot demonstrations. Together, these results demonstrate the potential of action flow as a shared control interface connecting heterogeneous training data, open-loop policy evaluation, and robot control.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 1.00
“Together, these results demonstrate the potential of action flow as a shared control interface connecting heterogeneous training data, open-loop policy evaluation, and robot control.”
How the main claim is stated: asserted 0.76
What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.08 · human participants 0.17 · controlled comparison 0.95 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.96 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| We introduce Hydra-0, a generalist world model conditioned on action flow, which represents robot actions as pixel motion. | contribution 0.98 | 0.91 | 0.08 | asserted 1.00 |
| This shared visual interface enables generalist world modeling and control by learning action consequences across embodiments, tasks, environments, and video-generation backbones. | interpretation 0.36 | 0.57 | 0.18 | asserted 1.00 |
| Our best configuration achieves 90.4% lower robot-motion error and 60.2% lower object-motion error than our action-conditioned baseline, while supporting zero-shot composition and data-efficient adaptation. | result 1.00 | 0.96 | 0.24 | asserted 1.00 |
| On the RoboLab benchmark, Hydra-0 achieves a Pearson correlation of r=0.96 between replayed and reference success rates. | result 1.00 | 0.96 | 0.09 | asserted 1.00 |
| Finally, we uncover an emergent inverse mode of this interface: a world action model that predicts compatible robot motion from desired object flow transferred from a human demonstration. | result 0.52 | 0.93 | 0.11 | asserted 1.00 |
| A trained action head maps the resulting latent features to executable actions without requiring task-specific expert robot demonstrations. | method 0.96 | 0.84 | 0.06 | asserted 1.00 |
| Together, these results demonstrate the potential of action flow as a shared control interface connecting heterogeneous training data, open-loop policy evaluation, and robot control. | interpretation 1.00 | 0.79 | 0.82 | asserted 0.76 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.90
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 5,567 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (11)
Showing 10 of the paper's 11 figures. Figures are the authors', reproduced from the paper.
References (67)
- [1] LightX2V: Light Video Generation Inference Framework, August 2026. URL https://gi thub.com/ModelTC/LightX2V . original-date: 2025-03-24T10:27:56Z on Pith
- [2] AgiBot-World-Contributors, Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, Shu Jiang, Yuxin Jiang, Cheng Jing, Hongyang Li, Jialu Li, Chiming Liu, Yi Liu, Yuxiang Lu, Jianlan Luo, Ping Luo, Yao Mu, Yuehan Niu, Yix... on PithPith's reviewarXiv
- [3] Arthur Allshire, Himanshu Gaurav Singh, Ritvik Singh, Adam Rashid, Hongsuk Choi, David McAllister, Justin Yu, Yiyuan Chen, Huang Huang, Pieter Abbeel, Xi Chen, Rocky Duan, Phillip Isola, Jitendra Malik, Fred Shentu, Guanya Shi, Philipp Wu, and Angjoo Kanazawa. Scalable Behavio... on PithPith's reviewarXiv
- [4] Sami Azirar, Enrico Pallotta, Jan Nogga, Jürgen Gall, Sven Behnke, and Hermann Blum. Contact- Flow: A video action conditioning that transfers across embodiments, July 2026. URLhttp:// arxiv.org/abs/2607.26579. arXiv:2607.26579 [cs.RO] on PithPith's reviewarXiv
- [5] Homanga Bharadhwaj, Roozbeh Mottaghi, Ab- hinav Gupta, and Shubham Tulsiani. Track2Act: Predicting Point Tracks from Internet Videos Enables Generalizable Robot Manipulation. In European Conference on Computer Vision, 2024. ISBN 978-3-031-73116-7. doi: 10.1007/978-3-031 -73116-7_18 on Pithdoi:10.1007/978-3-031
- [6] Kevin Black, Noah Brown, Danny Driess, Ad- nan Esmail, Michael Equi, Chelsea Finn, Nic- 14 Hydra-0 : Action Flow for Generalist World Modeling and Control colo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyim- ing Ke, Sergey Levine, Adrian Li... on PithPith's reviewarXiv
- [7] Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantaf... on PithPith's reviewarXiv
- [8] Tianxiao Chen, Jintao Rong, Huajin Chen, Jingya Wang, Tao Zhou, Jiming Chen, and Qi Ye. MIMIC: Mask-Injected Manipulation Video Gen- eration with Interaction Control. October 2025. URL https://openreview.net/forum?id=CO rUdVuInH on Pith
- [9] Yixiang Chen, Peiyan Li, Yuan Xu, Qisen Ma, Ji- abing Yang, Kai Wang, Jianhua Yang, Dong An, He Guan, Gaoteng Liu, Jianlou Si, Jun Huang, Jing Liu, Nianfeng Liu, Yan Huang, and Liang Wang. FlowWAM: Optical Flow as a Unified Action Representation for World Action Models, July 2... on PithPith's reviewarXiv
- [10] Yixiang Chen, Peiyan Li, Jiabing Yang, Keji He, Xiangnan Wu, Yuan Xu, Kai Wang, Jing Liu, Nianfeng Liu, Yan Huang, and Liang Wang. BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Masks, February 2026. URLhttp://arxiv.or g/abs/2602.03793. arX... on PitharXiv
- [11] Yukang Chen, Luozhou Wang, Wei Huang, Shuai Yang, Bohan Zhang, Yicheng Xiao, Ruihang Chu, Weian Mao, Qixin Hu, Shaoteng Liu, Yuyang Zhao, Huizi Mao, Ying-Cong Chen, Enze Xie, Xiaojuan Qi, and Song Han. LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation, M... on PithPith's reviewarXiv
- [12] Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal Ma- nipulation Interface: In-The-Wild Robot Teach- ing Without In-The-Wild Robots, February 2024. URL http://arxiv.org/abs/2402.10329 . arXiv:2402.10329 [cs] on PithPith's reviewarXiv
- [13] Ruihang Chu, Yefei He, Zhekai Chen, Shi- wei Zhang, Xiaogang Xu, Bin Xia, Dingdong Wang, Hongwei Yi, Xihui Liu, Hengshuang Zhao, Yu Liu, Yingya Zhang, and Yujiu Yang. Wan- Move: Motion-controllable Video Generation via Latent Trajectory Guidance, December 2025. URL http://arxi... on PitharXiv
- [14] Haoquan Fang, Jiafei Duan, Donovan Clay, Sam Wang, Shuo Liu, Weikai Huang, Xiang Fan, Wei-Chuan Tsai, Shirui Chen, Yi Ru Wang, Shanli Xing, Jaemin Cho, Jae Sung Park, Ainaz Eftekhar, Peter Sushko, Karen Far- ley, Angad Wadhwa, Cole Harrison, Winson Han, Ying-Chun Lee, Eli Vand... on PithPith's reviewarXiv
- [15] Jiahui Fu, Junyu Nan, Lingfeng Sun, Hongyu Li, Jianing Qian, Jennifer L. Barry, Kris Ki- tani, and George Konidaris. NovaPlan: Zero- Shot Long-Horizon Manipulation via Closed- Loop Video Language Planning, February 2026. URL http://arxiv.org/abs/2602.20119 . arXiv:2602.20119 [cs] on PithPith's reviewarXiv
- [16] Chongkai Gao, Haozhuo Zhang, Zhixuan Xu, Zhehao Cai, and Lin Shao. FLIP: Flow- Centric Generative Planning as General-Purpose Manipulation World Model, February 2025. URL http://arxiv.org/abs/2412.08261 . arXiv:2412.08261 [cs] on PithPith's reviewarXiv
- [17] Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Ta- tiana Lopez-Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, Chen Sun, Oliver Wang, An- drew Owens, and Deqing Sun. Motion Prompt- ing: Controlling Video Generation with Motion Tr... on PithPith's reviewarXiv
- [18] Nate Gillman, Yinghua Zhou, Zitian Tang, Evan Luo, Arjan Chakravarthy, Daksh Aggarwal, Michael Freeman, and Chen Sun. Goal Force: Teaching Video Models To Accomplish Physics- Conditioned Goals. pages 20077–20087, 2026. URL https://openaccess.thecvf.com/cont ent/CVPR2026/html/G... on Pith
- [19] Yanjiang Guo, Lucy Xiaoyang Shi, Jianyu Chen, and Chelsea Finn. Ctrl-World: A Controllable Generative World Model for Robot Manipulation, October 2025. URLhttp://arxiv.org/abs/25 10.10125. arXiv:2510.10125 [cs] on PithPith's reviewarXiv
- [20] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models.Nature, 640(8059): 647–653, April 2025. ISSN 1476-4687. doi: 10.1 038/s41586-025-08744-2. URL https://www.na ture.com/articles/s41586-025-08744-2 on Pith
show 47 more
- [21] Adam W. Harley, Yang You, Xinglong Sun, Yang Zheng, Nikhil Raghuraman, Yunqi Gu, Shel- don Liang, Wen-Hsuan Chu, Achal Dave, Pavel Tokmakov, Suya You, Rares Ambrus, Katerina Fragkiadaki, and Leonidas J. Guibas. AllTracker: Efficient Dense Point Tracking at High Resolu- tion, J... on PithPith's reviewarXiv
- [22] Martin Heusel, Hubert Ramsauer, Thomas Un- terthiner, Bernhard Nessler, and Sepp Hochre- iter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Sys- tems, volume 30. Curran Associates, Inc., 2017. U... on Pith
- [23] Ryan Hoque, Peide Huang, David J. Yoon, Mouli Sivapurapu, and Jian Zhang. EgoDex: Learning Dexterous Manipulation from Large-Scale Ego- centric Video, May 2025. URLhttp://arxiv. org/abs/2505.11709. arXiv:2505.11709 [cs] on PithPith's reviewarXiv
- [24] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, October on Pith
- [25] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Nic- colo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szy- mon Jakubczak, Tim Jones, Liyiming Ke, Devi... on PithPith's reviewarXiv
- [26] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Mill... on PithPith's reviewarXiv
- [27] Byungjun Kim, Taeksoo Kim, Hyunsoo Cha, and Hanbyul Joo. Robot-Factored World Models via Robot Rendering, July 2026. URLhttp://ar 16 Hydra-0 : Action Flow for Generalist World Modeling and Control xiv.org/abs/2607.22535 . arXiv:2607.22535 [cs.RO] version: 1 on PithPith's reviewarXiv
- [28] Gen Li, Bo Zhao, Jianfei Yang, and Laura Sevilla- Lara. Mask2IV: Interaction-Centric Video Gen- eration via Mask Trajectories, November 2025. URL http://arxiv.org/abs/2510.03135 . arXiv:2510.03135 [cs] on PitharXiv
- [29] Hongyu Li, Lingfeng Sun, Yafei Hu, Duy Ta, Jennifer Barry, George Konidaris, and Jiahui Fu. NovaFlow: Zero-Shot Manipulation via Action- able Flow from Generated Videos, October 2025. URL http://arxiv.org/abs/2510.08568 . arXiv:2510.08568 [cs] on PitharXiv
- [30] Hongyu Li, Wanjia Fu, Xiaoyan Cong, Zekun Li, Binghao Huang, Hanxiao Jiang, Xintong He, Yiqing Liang, Rao Fu, Tao Lu, Srinath Sridhar, Kevin A. Smith, George Konidaris, and Yunzhu Li. Deform360: A Massive Multi-view Visuotac- tile Dataset for Deformable World Models, July on Pith
- [31] Shaowei Liu, Xuanchi Ren, Tianchang Shen, Huan Ling, Saurabh Gupta, Shenlong Wang, Sanja Fidler, and Jun Gao. MoRight: Motion Control Done Right, April 2026. URLhttp:// arxiv.org/abs/2604.07348. arXiv:2604.07348 [cs] on PithPith's reviewarXiv
- [32] Wei Liu, Ziyu Chen, Zizhang Li, Yue Wang, Hong-Xing Yu, and Jiajun Wu. RealWonder: Real-Time Physical Action-Conditioned Video Generation, March 2026. URL http://arxiv. org/abs/2603.05449. arXiv:2603.05449 [cs] on PitharXiv
- [33] NVIDIA, Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi "Jim" Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop,... on PithPith's reviewarXiv
- [34] NVIDIA, Mayank Mittal, Pascal Roth, James Tigue, Antoine Richard, Octi Zhang, Peter Du, Antonio Serrano-Muñoz, Xinjie Yao, René Zurbrügg, Nikita Rudin, Lukasz Wawrzyniak, Milad Rakhsha, Alain Denzler, Eric Heiden, Ales Borovicka, Ossama Ahmed, Iretiayo Aki- nola, Abrar Anwar, ... on PithPith's reviewarXiv
- [35] NVIDIA, Aditi, Niket Agarwal, Arslan Ali, Jon Allen, Martin Antolini, Adeline Aubame, Alis- son Azzolini, Junjie Bai, Maciej Bala, Yogesh Balaji, Josh Bapst, Aarti Basant, Mukesh Be- ladiya, Mohammad Qazim Bhat, Zaid Pervaiz Bhat, Dan Blick, Vanni Brighella, Han Cai, Tiffany C... on PithPith's reviewarXiv
- [36] NVIDIA, Arslan Ali, Junjie Bai, Maciej Bala, Yogesh Balaji, Aaron Blakeman, Tiffany Cai, Jiaxin Cao, Tianshi Cao, Elizabeth Cha, Yu- Wei Chao, Prithvijit Chattopadhyay, Mike Chen, Yongxin Chen, Yu Chen, Shuai Cheng, Yin Cui, Jenna Diamond, Yifan Ding, Jiaojiao Fan, Linxi Fan, ... on PithPith's reviewarXiv
- [37] Julian Hector Quevedo, Ansh Kumar Sharma, 18 Hydra-0 : Action Flow for Generalist World Modeling and Control Yixiang Sun, Varad Suryavanshi, Percy Liang, and Sherry Yang. WorldGym: World Model as An Environment for Policy Evaluation. October on Pith
- [38] YuHang Tang, Yixuan Lou, Pengfei Han, Haom- ing Song, Xinyi Ye, Dong Wang, and Bin Zhao. Trajectory Conditioned Cross-embodiment Skill Transfer, October 2025. URLhttp://arxiv.or g/abs/2510.07773. arXiv:2510.07773 [cs] on PitharXiv
- [39] Gemini Robotics Team, Krzysztof Choroman- ski, Coline Devin, Yilun Du, Debidatta Dwibedi, Ruiqi Gao, Abhishek Jindal, Thomas Kipf, Sean Kirmani, Isabel Leal, Fangchen Liu, Anirudha Majumdar, Andrew Marmon, Carolina Parada, Yulia Rubanova, Dhruv Shah, Vikas Sindhwani, Jie Tan, ... on PitharXiv
- [40] Gemma Team, Sherif El Abd, Vaibhav Aggar- wal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, Mayank Chaturvedi, Aditya Chawla, Victor Cotruta, Alice Coucke, Phil Culliton, Robert Dadashi, Lucas Dixon, Mohamed Elhaw... on PithPith's reviewarXiv
- [41] Wei-Cheng Tseng, Jinwei Gu, Qinsheng Zhang, Hanzi Mao, Ming-Yu Liu, Florian Shkurti, and Lin Yen-Chen. Scalable Policy Evalua- tion with Video World Models, December 2025. URL http://arxiv.org/abs/2511.11520 . arXiv:2511.11520 [cs] on PitharXiv
- [42] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michal- ski, and Sylvain Gelly. Towards Accurate Gen- erative Models of Video: A New Metric & Chal- lenges, March 2019. URLhttp://arxiv.org/ abs/1812.01717. arXiv:1812.01717 [cs.CV] on PithPith's reviewarXiv
- [43] Homer Rich Walke, Kevin Black, Tony Z. Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, Abraham Lee, Kuan Fang, Chelsea Finn, and Sergey Levine. BridgeData V2: A Dataset for Robot Learning at Scale. InPro- ceedings of ... on Pith
- [44] Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, JiayuWang, JingfengZhang, JingrenZhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng ... on PithPith's reviewarXiv
- [45] Angtian Wang, Haibin Huang, Jacob Zhiyuan Fang, Yiding Yang, and Chongyang Ma. ATI: Any Trajectory Instruction for Controllable Video Generation, June 2025. URL http:// arxiv.org/abs/2505.22944. arXiv:2505.22944 [cs] on PithPith's reviewarXiv
- [46] Ying Wang, Oumayma Bounou, Yann LeCun, and Mengye Ren. AdaJEPA: An Adaptive La- tent World Model, June 2026. URLhttp://ar xiv.org/abs/2606.32026 . arXiv:2606.32026 [cs.LG] on PithPith's reviewarXiv
- [47] Ying Wang, Oumayma Bounou, Gaoyue Zhou, Randall Balestriero, Tim G. J. Rudner, Yann Le- Cun, and Mengye Ren. Temporal Straightening for Latent Planning, June 2026. URLhttp:// arxiv.org/abs/2603.12231. arXiv:2603.12231 [cs.LG] on PithPith's reviewarXiv
- [48] Yixuan Wang, Rhythm Syed, Fangyu Wu, Mengchao Zhang, Aykut Onol, Jose Barreiros, Hooshang Nayyeri, Tony Dear, Huan Zhang, and Yunzhu Li. Interactive World Simulator for Robot Policy Training and Evaluation, March on Pith
- [49] Zhuoyuan Wu and Jun Gao. OSCAR: Omni- Embodiment Action-Conditioned World Model for Robotics, June 2026. URLhttp://arxiv.or g/abs/2606.04463. arXiv:2606.04463 [cs.RO] on PithPith's reviewarXiv
- [50] Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Wetzstein, Manuela Veloso, and Shuran Song. Flow as the Cross-domain Ma- nipulation Interface. InConference on Robot Learning, January 2025. URLhttps://procee dings.mlr.press/v270/xu25a.html on Pith
- [51] Mutian Xu, Tianbao Zhang, Tianqi Liu, Zhaoxi Chen, Xiaoguang Han, and Ziwei Liu. Kinema4D: 20 Hydra-0 : Action Flow for Generalist World Modeling and Control Kinematic 4D World Modeling for Spatiotempo- ral Embodied Simulation, March 2026. URL h t tp : / / a r x i v . o r g / ... on PitharXiv
- [52] Xuning Yang, Rishit Dagli, Alex Zook, Hugo Hadfield, Ankit Goyal, Stan Birchfield, Fabio Ramos, and Jonathan Tremblay. RoboLab: A High-Fidelity Simulation Benchmark for Anal- ysis of Task Generalist Policies, April 2026. URL http://arxiv.org/abs/2604.09860 . arXiv:2604.09860 [cs] on PithPith's reviewarXiv
- [53] Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T. Freeman. Improved Distribution Matching Distillation for Fast Image Synthesis, May 2024. URL http://arxiv.org/abs/2405 .14867. arXiv:2405.14867 [cs.CV] on PithPith's reviewarXiv
- [54] URL http://arxiv.org/abs/2603.085 on Pith
- [55] arXiv:2603.08546 [cs] on PitharXiv
- [56] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In2018 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 586– 595, June 2018. doi: 10.1109/CVPR.2018.000... on PitharXiv
- [57] Jinliang Zheng, Jianxiong Li, Zhihao Wang, Dongxiu Liu, Xirui Kang, Yuchun Feng, Yi- nan Zheng, Jiayin Zou, Yilun Chen, Jia Zeng, Ya-Qin Zhang, Jiangmiao Pang, Jingjing Liu, Tai Wang, and Xianyuan Zhan. X-VLA: Soft-Prompted Transformer as Scalable Cross- EmbodimentVision-Langu... on PithPith's reviewarXiv
- [58] Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World Models on Pre- trained Visual Features enable Zero-shot Plan- ning. InInternational Conference on Machine Learning, June 2025. URLhttps://openrevi ew.net/forum?id=D5RNACOZEI on Pith
- [59] Hongzhou Zhu, Min Zhao, Guande He, Hang Su, Chongxuan Li, and Jun Zhu. Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation, February 2026. URLhttp://arxi v.org/abs/2602.02214. arXiv:2602.02214 [cs] on PithPith's reviewarXiv
- [61] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T. Free- man, and Taesung Park. One-step Diffusion with Distribution Matching Distillation, October 2024. URL http://arxiv.org/abs/2311.18828 . arXiv:2311.18828 [cs] on PithPith's reviewarXiv
- [62] Tianwei Yin, Qiang Zhang, Richard Zhang, William T. Freeman, Fredo Durand, Eli Shecht- man, and Xun Huang. From Slow Bidirectional to Fast Autoregressive Video Diffusion Models, September 2025. URLhttp://arxiv.org/abs/ 2412.07772. arXiv:2412.07772 [cs] on PitharXiv
- [67] Appendix 8.1. Additional Qualitative Evaluation Additional qualitative results on DROID are shown in fig. 11. 8.2. Action Flow Construction Details For metadata-rich robot data and deployment, we sample points on the visible robot surface, propagate them using robot link trans... on Pith
- [85] arXiv:2106.09685 [cs.CL] on PithPith's reviewarXiv
- [90] arXiv:2607.05390 [cs.RO] on PithPith's reviewarXiv
- [2021] URL http://arxiv.org/abs/2106.096 on Pith
- [2025] URL https://openreview.net/forum?i d=hidBHy1CAw on Pith
- [2026] URL http://arxiv.org/abs/2607.053 on Pith
Cited by 4 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining cs.CV · Oct 2026
- ThinkWorld: World Imagination Reinforces Chain-of-Dynamics Reasoning cs.RO · Oct 2026
- Dream4ACT: A Shared Visual Action Interface for Multi-Embodiment Video-Action Modeling cs.RO · Sep 2026
- Seeing, Maintaining, and Learning: The Evolution of Multimodal Agent Memory cs.AI · Oct 2026
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites · 32 sentences without a confident label
Builds on 2 papers
-
Building on trajectory-based visual conditioning in ATI and Wan-Move [ref] , our world model uses this condition to predict the command's consequences, a mode we call kinematically grounded video prediction.
ATI: Any Trajectory Instruction for Controllable Video Generation · Introduction · 0.85
-
Building on trajectory-based visual conditioning in ATI and Wan-Move [ref] , our world model uses this condition to predict the command's consequences, a mode we call kinematically grounded video prediction.
2512.08765 · Introduction · 0.85
Uses the method or tool of 5 papers
-
Following LongLive-2.0 [ref] , we initialize the distilled student from the causal autoregressive checkpoint and generate each chunk with four denoising steps through its causal cache while preserving the same absolute-offset motion slices.
LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation · Efficient Causal Rollout · 0.94
-
For efficient robotics deployment, we combine autoregressive conversion for causal long-horizon generation with improved distribution-matching distillation (DMD2) [ref] for few-step sampling.
One-step Diffusion with Distribution Matching Distillation · Efficient Causal Rollout · 1.00
-
For efficient robotics deployment, we combine autoregressive conversion for causal long-horizon generation with improved distribution-matching distillation (DMD2) [ref] for few-step sampling.
Improved Distribution Matching Distillation for Fast Image Synthesis · Efficient Causal Rollout · 1.00
Uses data or values from 6 papers
-
We use six tasks from the Interactive World Simulator (IWS) dataset [ref] : Bimanual box, Bimanual rope, Bimanual sweep, Push-T, Single chain-in-box, and Single grasp.
2603.08546 · Data Efficiency from Multi-Embodiment Mid-Training · 0.99
-
EgoDex [ref] provides egocentric human-hand manipulation captured with Apple Vision Pro, while Deform360 [ref] provides handheld-gripper demonstrations of deformable-object manipulation using UMI grippers [ref] .
Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots · Datasets · 0.96
-
EgoDex [ref] provides egocentric human-hand manipulation captured with Apple Vision Pro, while Deform360 [ref] provides handheld-gripper demonstrations of deformable-object manipulation using UMI grippers [ref] .
EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video · Datasets · 0.96
Cites as background 13 papers
-
Action-conditioned video simulators have been used for policy evaluation and ranking, synthetic policy-improvement data, and interactive rollouts: [ref] and [ref] study policy ranking; [ref] and [ref] also use imagined trajectories for policy improvement or training; and Veo [ref] studies nominal and out-of-distribution policy behavior.
2511.11520 · Neural Simulators and Robotic World Models · 0.84
-
Action-conditioned video simulators have been used for policy evaluation and ranking, synthetic policy-improvement data, and interactive rollouts: [ref] and [ref] study policy ranking; [ref] and [ref] also use imagined trajectories for policy improvement or training; and Veo [ref] studies nominal and out-of-distribution policy behavior.
2512.10675 · Neural Simulators and Robotic World Models · 0.84
-
Action-conditioned video simulators have been used for policy evaluation and ranking, synthetic policy-improvement data, and interactive rollouts: [ref] and [ref] study policy ranking; [ref] and [ref] also use imagined trajectories for policy improvement or training; and Veo [ref] studies nominal and out-of-distribution policy behavior.
2603.08546 · Neural Simulators and Robotic World Models · 0.84
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment