Pith.

Robotics · arXiv:2610.12099 v1 · submitted 8 Oct 2026 · 82 references

UNITAS: A 3D-Native World Action Model for Embodied Manipulation

Ruixiang Wang, Yongyi Su, Wenlve Zhou, Bo Yue, Hengyan Liu, Dekun Lu, Yuxin Tian, Yihan Fang

show 9 more authors

Zerui Wu, Xing Hu, Jietao Chen, Yong Guo, Ziyan He, Junbin Yuan, Guiliang Liu, Xiaofen Xing, Kui Jia

The paper says

“World action models (WAMs) aim to answer a coupled physical question: given a task instruction, what motion should the robot execute, and how will that motion change the surrounding world?”

more of the abstract

Most existing WAMs build on pretrained video generators and represent world evolution through images or visual latents. Robotic interaction, however, takes place in metric three-dimensional space, while images are view-dependent projections whose pixel distances do not directly encode physical distances. We introduce UNITAS, to our knowledge the first 3D-native world action model that unifies observations, actions, and scene dynamics in a shared metric 3D frame within each interaction, using a common representation across robot embodiments and human hands. Action flow represents human hands and robot grippers as 3D point trajectories, while scene flow describes scene-point displacements conditioned on these trajectories. World-aligned 3D positional embeddings ground visual tokens with or without depth input, and a physical-time trajectory tokenizer encodes each point trajectory as one token anchored at its current 3D position. This interface supports both direct action execution and action-conditioned scene prediction. With 1.7B parameters, UNITAS achieves the best action-conditioned scene prediction on RoboTwin among the compared methods, with up to 49% lower displacement errors than PointWorld, and state-of-the-art manipulation success, including 99.8% on LIBERO and an average of 85% across real-world tasks. The code is available at https://github.com/DexForce/UNITAS.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 10 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A clean, well-scoped 3D-native action-and-world model at 1.7B that deserves serious refereeing, with one internal inconsistency and one coupling assumption that need fixes before the headline claim is fully settled. This line is wrong

3 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 82 references say.

The paper says, at Sec. 1 vs Table 2

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The introduction reports real-robot success of 90% on Organize Books and 65% on Pack the Ball, but Table 2 reports UNITAS at 16/20 = 80% on Pack the Ball; the 65% figure is exactly Fast-WAM's Organize Books score in the same table.

more of objection 1

The abstract's 'average of 85% across real-world tasks' matches (90+80)/2 and not (90+65)/2, so Table 2 and the abstract agree and the introduction is the outlier. This internal inconsistency must be corrected because the real-robot performance is part of the headline quantitative claim.

The paper says, at Sec. 4.3, Table 2

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The real-robot evaluation uses 20 trials per method per task with no error bars, confidence intervals, or seeds. At n=20, the 95% binomial confidence interval for UNITAS on Pack the Ball (16/20) is roughly [56%, 94%], which overlaps the intervals for pi0.5 (11/20) and Fast-WAM (12/20); the Organize Books comparison (18/20 vs 15/20 and 13/20) is likewise consistent with substantial chance variation.

more of objection 2

The claim that UNITAS outperforms both baselines on these tasks is therefore not established at the reported scale, and the same absence of variance/seed information affects the near-saturated LIBERO numbers in Table 1, where differences such as 99.8% vs 98.1% need episode counts or confidence intervals to interpret.

The paper says, at Eq. (3), Sec. 3.5, Appendix F.2

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The factorization p(A,u,S|C,q0) = p(A|C) p(u|C,q0) p(S|A,C) makes scene prediction independent of the executable command chunk u given the action flow A, and the attention structure in Sec. 3.5 ensures action-chunk tokens never attend to action-flow tokens while A and u are denoised in parallel with independent noise.

more of objection 3

Since policy execution runs u while simulator mode conditions on A, nothing in the training objective (Eq. 9) or the architecture enforces that the predicted scene follows the executed motion. The closed-loop consistency test in Appendix F.2 is the right check and the authors transparently call it 'indirect evidence'; however, it covers only 3 of the 50 RoboTwin tasks, and its Moving Acc@1cm (64.26%) and Moving ADE (1.211 cm) are far below the open-loop Table 4 numbers (92.31%, 0.427 cm) on a different task set, so the gap cannot be attributed to task difficulty versus A-u divergence. A matched-task comparison (scoring the same tasks both open-loop and closed-loop, or running the closed-loop protocol on the full task set) would resolve whether the world-model predictions describe the consequences of the motion the robot actually executes; this is load-bearing for the 'world action' claim.

4 minor objections

The paper says, at Sec. 4.3

Pith says: minor objection 1 of 4

The real-robot experimental setup is not described: the paper names the two tasks and the trial count but does not state the robot arm, gripper, camera configuration, or task-success criteria, which are needed to reproduce the 20-trial evaluation.

The paper says, at Sec. 5, Fig. 5

Pith says: minor objection 2 of 4

The scene-flow visualizations in Fig. 5 and Appendix G.1 are drawn from training clips, as the paper itself notes in Sec. 5 and in 'these examples illustrate learned geometry rather than held-out accuracy'; adding at least one held-out visualization in the main text would let the reader judge generalization from the figure itself.

show 2 more

The paper says, at Appendix B.1

Pith says: minor objection 3 of 4

The depth-removal evaluation is reported at 100K training steps, while the main LIBERO-Plus results in Table 1 are reported at the final checkpoint; stating the checkpoint step associated with each LIBERO-Plus row would prevent confusion when comparing Tables 1, 3, 9, and 10.

The paper says, at Title/header

Pith says: minor objection 4 of 4

The running header renders the title as '3D-NATIVEWORLDACTIONMODEL' and 'EMBODIEDMANIPULATION' with missing spaces, which should be fixed in the camera-ready source.

What would settle it

Extend Appendix F.2 to all 50 RoboTwin tasks. For each sampled (A,u), execute u, record ground-truth scene S_gt(u); run simulator mode conditioned on sampled A (S_pred(A)) and on the gripper trajectories recovered by forward kinematics from the executed u (S_pred(u)). Compare Moving ADE/FDE and Acc@1cm of both against S_gt(u) using F.1's scoring. If S_pred(u) beats S_pred(A) by more than roughly 5% ADE or 5 Acc points, Eq. 3's independence fails in deployment and the model simulates a motion the robot does not execute. If S_pred(A) is statistically indistinguishable from S_pred(u) and the 50-task Acc@1cm is comparable to the reported 64.26%, the factorization is supported. Report confidence intervals or seeds: the current consistency claim rests on 60 trials.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 10 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 1 min 11 s · 26,663 tokens read, 8,987 written
  3. Editor wrote the verdict above 37 s · 27,865 tokens read, 4,275 written
  4. Referee major revision: 3 major objections, 4 minor 1 min 52 s · 27,072 tokens read, 13,615 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Read UNITAS (2610.12099). It deserves a serious referee. The core idea is coherent: observations, gripper/hand trajectories, and scene response live in one metric frame, with a physical-time tokenizer that makes mixed frame-rate data tractable. The genuinely new piece is joint control and consequence prediction in the same model at 1.7B. PointWorld does metric scene flow but only from externally supplied motion, and the video WAMs stay in pixel space. The physical-time tokenizer (one token per trajectory, time in seconds) is the quietly useful idea here. There is also real diligence: the depth-free grounding router is evaluated separately, the qualitative scene figures are flagged as training clips, and real-world scene prediction is explicitly labeled as outside scene-flow supervision. Code is out. The closed-loop action–scene consistency test in Appendix F.2 is a good-faith attempt to check the coupling, not a dodge.

Now the soft spots, in proportion.

One, the Pack the Ball discrepancy is real: the intro reports 65%, Table 2 reports 16/20, and the abstract's 85% average matches (90+80)/2, so the intro number looks stale rather than fabricated. But it is exactly the kind of inconsistency that makes a reader re-check every other number. Easy fix, must fix.

Two, no seeds or error bars on benchmarks that are near saturation (99.8% LIBERO). A single run at that level is fragile evidence. Common in the field, still worth demanding.

Three, the load-bearing assumption in Eq. 3: scene flow conditions on sampled action flow A only, while the executed command u is sampled in parallel with independent noise, and the attention mask keeps u from seeing A. At training time both come from the same demonstration; at inference they can diverge, and the world model then predicts the consequences of a motion the robot does not execute. The closed-loop test is the only check and covers three RoboTwin tasks (64.26% Moving Acc@1cm vs 92.31% open-loop; task sets differ, so the gap is suggestive). The authors call it indirect evidence — fair — but the "world action model" claim rests on exactly this coupling, so the review should ask for a full-distribution comparison of scene prediction conditioned on A versus on the executed u.

Four, the video-baseline lifting protocol (SpatialTrackerV2 plus predicted depth, calibrated on initial-frame ground-truth depth) gives UNITAS a systematic comparison advantage in Table 4. The protocol is disclosed and reasonable, but the comparison is partly apples-to-oranges regardless of calibration.

The central design holds up; these are additive fixes, not a sunk foundation. Send it to review. If the real-robot numbers get reconciled and the consistency test is broadened, it is an accept-shaped paper.

The strongest objection

The central claim — that UNITAS is a world action model whose scene dynamics describe the consequences of the robot's motion in a shared metric frame — requires that the scene prediction follow the motion the robot actually executes. Eq. 3 factorizes p(A,u,S|C,q0)=p(A|C)p(u|C,q0)p(S|A,C), so S is independent of u given A, and A and u are sampled in parallel with independent noise; Sec. 3.5 confirms action-chunk tokens do not attend to action-flow tokens. A and u are different parameterizations of the same physical motion: A is 20 URDF gripper point trajectories, u is an end-effector command chunk. Nothing in the training objective forces their sampled outputs to be consistent; the only empirical check is the closed-loop action–scene consistency test in Appendix F.2, on 3 of 50 RoboTwin tasks, 10 trials per setting. Under this test, with UNITAS's own generated action flow conditioning scene prediction while the separately sampled command is executed, Moving Acc@1cm is 64.26% and Moving ADE is 1.211 cm, substantially lower than the Table 4 open-loop numbers (92.31%, 0.427 cm) on the broader 50-task set; the task sets differ, so the gap is suggestive rather than conclusive, which is exactly why the broader test is needed. If the gap reflects divergence between A and u, the world model predicts the consequences of a motion the robot does not execute, and the 'world action' coupling in Eq. 3 fails where it matters. The paper is transparent that consistency is 'indirect evidence' and that broader scene metrics use externally supplied trajectories, but transparency does not close the gap: the assumption needs a test spanning the full task distribution and a direct comparison of scene predictions conditioned on A versus conditioned on the executed u. This is a correctness risk, not a novelty or consensus dispute.

The reader's weakest_assumption and my concern coincide: the load-bearing link is Eq. 3, with scene prediction conditioned on A alone while u is sampled in parallel. I checked the architecture to make sure this is not a straw man. Sec. 3.5 states that action-chunk tokens attend only to observation/language keys and values, the current end-effector state, and one another, not to action-flow tokens; the only coupling between A and u is shared context and a shared demonstration target. Since both predict the same physical motion in different parameterizations, the independence could hold approximately; the question is empirical, and the evidence is thin. Appendix F.2 is exactly the right experiment, but it covers 3 of 50 tasks, 10 trials per setting, and it conflates scene-prediction error on self-generated flow with divergence between A and u. My proposed decomposition separates those two error sources. I do not elevate the other reader concerns to primary status. The 65% vs 80% Pack the Ball contradiction in the introduction versus Table 2 is real, but it is a reporting inconsistency that can be corrected without affecting the method's structure. Missing seeds and error bars weaken comparative claims but do not bear directly on whether Eq. 3 holds. The video-lifting protocol in Table 4 may inflate UNITAS's margin over video baselines, but it does not threaten the absolute scene-prediction result. Qualitative-only real-world scene prediction is explicitly acknowledged in Sec. G.2 and Fig. 11, so it is a stated limitation rather than a hidden flaw. Credit is due where the paper earns it: the policy results are externally benchmarked, the code is released, the ablations are clean, and Appendix F.2 is an honest attempt to test the weak link instead of avoiding it. None of that removes the need for a broader consistency test. Because this is the same soft spot the reader already identified, my pass does not change the CONDITIONAL verdict; it sharpens the condition that should be attached: demonstrate A/u consistency over the full RoboTwin suite, or condition scene prediction on the executed command and show the closed-loop metrics hold.

The referee's report

If the results hold, this is a meaningful advance: it would demonstrate that a metric 3D trajectory interface can match or exceed much larger video-based world action models in both control and scene prediction at 1.7B parameters, supporting the paper's core thesis that interaction geometry is an effective shared substrate for world modeling and control. The evaluation has real strengths: headline results are measured against external benchmarks (LIBERO, RoboTwin 2.0, VLABench, LIBERO-Plus) and external baselines, so the central claim is not manufactured by the paper's own equations; the scene-prediction comparison in Table 4 shares initial queries, point identities, and motion conditions across methods; and the appendix includes per-task RoboTwin tables, a held-out tokenizer reconstruction study, a depth-free grounding evaluation, and a closed-loop consistency test. The main risk is whether the predicted scene dynamics describe the consequences of the motion the robot actually executes, since Eq. (3) decouples scene prediction from the executable command chunk; the paper's own closed-loop evidence bears on this but covers only three tasks.

Pith's reading of the paper (a machine's summary, not the authors')

UNITAS proposes a single answer to two coupled questions a robot must answer — what motion to execute, and how that motion will change the world — by expressing both in the same metric 3D frame, as point trajectories, rather than as future images. The paper's claim is that this 3D-native interface is sufficient to reach the current state of the art in both robot control and world modeling: at 1.7B parameters it reports 99.8% success on LIBERO, 92.94%/93.04% on RoboTwin 2.0, 89.1% on LIBERO-Plus, 56.2% on VLABench, and 90%/80% on two real-robot tasks, plus the lowest action-conditioned scene prediction error among the compared methods on RoboTwin. The reason to care is that if the claim holds, the expensive video-prediction machinery of recent world action models can be replaced by a compact, physically grounded trajectory representation shared across robot arms, human hands, and scenes, at a fraction of the parameter count.

What carries the argument. The device that carries the argument is the shared metric frame together with three mechanisms inside it. A fixed frame $\mathcal{F}_W$ is held constant within each prediction window; visual tokens receive world-aligned 3D positional embeddings computed by back-projecting depth, or, without depth, by a learned router that scores 64 candidate depths along each camera ray; and the physical-time trajectory tokenizer encodes each point trajectory — gripper, hand, or scene point — as a single latent token anchored at the point's current 3D position, using elapsed seconds $\tau_k = k/f$ rather than frame indices so that datasets of different frame rates align. A coupled Mixture-of-Transformers with two diffusion experts implements the factorization: a point dynamics expert denoises action- and scene-flow tokens while an action expert denoises the executable command chunk, with scene tokens attending only to clean action flow, so that scene prediction conditions on motion without feeding back into it, and policy-only inference can skip both flow branches entirely.

Core claim. The paper's central claim is that a world action model can be 3D-native: instead of representing observations and future states as images or visual latents, UNITAS expresses the interacting body's motion as action flow, the environment's response as scene flow, and both as 3D point trajectories in one metric frame per prediction window, with a common representation for robot grippers and human hands. The joint distribution factorizes as $p(A,u,S \mid C,q_0) = p(A \mid C)\,p(u \mid C,q_0)\,p(S \mid A,C)$, so the action flow and the executable command chunk are sampled in parallel and scene prediction conditions on the action flow alone. On this basis the paper reports, at 1.7B parameters, 99.8% average success on LIBERO, 92.94%/93.04% on RoboTwin 2.0 Clean and Randomized, 89.1% on LIBERO-Plus, 56.2% success rate on VLABench, 90%/80% on two real-robot tasks, and the best action-conditioned scene prediction on RoboTwin, with moving ADE 0.427 cm versus 0.656 cm for PointWorld and 49% lower static ADE. The intended conclusion is that physical interaction geometry itself — not video prediction and not model scale — can serve as the unifying substrate for embodied learning.

The premise it rests on. The load-bearing premise is the factorization $p(A,u,S \mid C,q_0) = p(A \mid C)\,p(u \mid C,q_0)\,p(S \mid A,C)$: that the scene response is fully determined by the generated action flow and that this flow stays consistent with the separately sampled executable command chunk, a premise the paper checks only on three RoboTwin tasks, with scene prediction itself quantified only in simulation.

If the paper is right (Pith's inferences, not the authors')

A policy need not predict pixels to act or to foresee consequences: UNITAS' policy-only mode generates executable commands directly, and its simulator mode predicts the scene's response to the same action flow, so one model covers both control and world modeling.

Video generation can be dropped from a world action model without sacrificing benchmark performance: at 1.7B parameters UNITAS reports 99.8% on LIBERO and beats the 6B-parameter Fast-WAM and 8B-parameter Motus on RoboTwin 2.0.

Spatial grounding survives missing depth: with depth removed at inference, the learned router over candidate depths keeps LIBERO-Plus success at 84.43% versus 84.68% with ground-truth depth.

Mixed-dataset pretraining over real robots, simulation, and human hands transfers positively: keeping the RoboTwin batch size fixed while adding other sources lowers RoboTwin error, raises LIBERO-Plus robustness under robot initial-state changes from 84.8% to 94.2%, and improves overall success by 2.6 percentage points.

Extending beyond the paper: the same trajectory-tokenizer interface could absorb full-body humanoid or multi-agent motion, since each body part becomes one anchored token, removing the need for embodiment-specific action formats entirely.

Extending beyond the paper: conditioning scene prediction on the executed command chunk as well as the action flow, or adding a consistency loss between them, would directly attack the paper's main residual risk, since its closed-loop consistency test covers only three RoboTwin tasks.

Extending beyond the paper: the millimeter-scale simulation scene errors make the model a candidate for pre-execution motion validation on real hardware, but that use would require a quantitative real-world scene-prediction benchmark, which the paper does not yet provide.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

World action models (WAMs) aim to answer a coupled physical question: given a task instruction, what motion should the robot execute, and how will that motion change the surrounding world? Most existing WAMs build on pretrained video generators and represent world evolution through images or visual latents. Robotic interaction, however, takes place in metric three-dimensional space, while images are view-dependent projections whose pixel distances do not directly encode physical distances. We introduce UNITAS, to our knowledge the first 3D-native world action model that unifies observations, actions, and scene dynamics in a shared metric 3D frame within each interaction, using a common representation across robot embodiments and human hands. Action flow represents human hands and robot grippers as 3D point trajectories, while scene flow describes scene-point displacements conditioned on these trajectories. World-aligned 3D positional embeddings ground visual tokens with or without depth input, and a physical-time trajectory tokenizer encodes each point trajectory as one token anchored at its current 3D position. This interface supports both direct action execution and action-conditioned scene prediction. With 1.7B parameters, UNITAS achieves the best action-conditioned scene prediction on RoboTwin among the compared methods, with up to 49% lower displacement errors than PointWorld, and state-of-the-art manipulation success, including 99.8% on LIBERO and an average of 85% across real-world tasks. The code is available at https://github.com/DexForce/UNITAS.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-09

The authors' main claim, as the classifier picked it out 0.93

“With 1.7B parameters, UNITAS achieves the best action-conditioned scene prediction on RoboTwin among the compared methods, with up to 49% lower displacement errors than PointWorld, and state-of-the-art manipulation success, including 99.8% on LIBERO and an average of 85% across real-world tasks.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.96 · states limitation 0.07 · human participants 0.13 · controlled comparison 0.79 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.99

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
World action models (WAMs) aim to answer a coupled physical question: given a task instruction, what motion should the robot execute, and how will that motion change the surrounding world? background 1.00 0.05 0.03 asserted 0.99
Most existing WAMs build on pretrained video generators and represent world evolution through images or visual latents. background 0.99 0.03 0.02 asserted 1.00
Robotic interaction, however, takes place in metric three-dimensional space, while images are view-dependent projections whose pixel distances do not directly encode physical distances. gap 0.68 0.05 0.03 asserted 1.00
We introduce UNITAS, to our knowledge the first 3D-native world action model that unifies observations, actions, and scene dynamics in a shared metric 3D frame within each interaction, using a common representation across robot embodiments and human hands. contribution 1.00 0.92 0.30 asserted 0.99
Action flow represents human hands and robot grippers as 3D point trajectories, while scene flow describes scene-point displacements conditioned on these trajectories. method 0.99 0.81 0.05 asserted 1.00
World-aligned 3D positional embeddings ground visual tokens with or without depth input, and a physical-time trajectory tokenizer encodes each point trajectory as one token anchored at its current 3D position. method 1.00 0.88 0.04 asserted 1.00
This interface supports both direct action execution and action-conditioned scene prediction. method 0.55 0.64 0.06 asserted 0.99
With 1.7B parameters, UNITAS achieves the best action-conditioned scene prediction on RoboTwin among the compared methods, with up to 49% lower displacement errors than PointWorld, and state-of-the-art manipulation success, including 99.8% on LIBERO and an average of 85% across real-world tasks. result 1.00 0.96 0.61 asserted 1.00
The code is available at https://github.com/DexForce/UNITAS. contribution 0.50 0.36 0.02 asserted 0.89

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.82

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-09: 1 request, 7,051 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (11)
Figure 1 of the paper
Figure 1. UNITAS is a 3D-native world action model. Observations, action flow, and scene flow share a metric frame within each prediction window, with a common representation across embod￾iments. With 1.7B parameters, UNITAS achieves leading manipulation results among the compared methods and predicts action-conditioned scene trajectories. Recent methods incorporate depth, optical flow, scene flow, or reconstructed 4D geometry into pre￾diction (Zhen et al., 2025; Guo et al., 2026; Zhao et al., 2026; Wang et al., 2026b) or ground action generation in explicit 3D structure (Shridhar et al.,…
Figure 2 of the paper
Figure 2. UNITAS architecture and attention masks. Left: The point dynamics expert models both action and scene flows, while the action expert generates the end-effector action chunk. Scene flow generation is conditioned on clean demonstrated action flow during training and generated or supplied action flow at inference. Right: Colored/white cells indicate allowed/blocked attention. Policy-only inference omits both future action-flow and scene-flow tokens. 3.1 REVISITING VIDEO WORLD ACTION MODELS Let o denote current image observations, ℓ the instruction, a1:H an action chunk, and v1:T future…
Figure 3 of the paper
Figure 3. Shared point-flow representations across data sources. Left: gripper/hand trajectories (purple) and scene trajectories (yellow–blue), with RGB insets. Right: RoboTwin end-effector XYZ error (mm). Thick curves show exponential moving averages (EMA) of the thin curves. pools the trajectory into one latent, and the decoder reads that latent back at a requested time: zn = 1 √ H X H k=1 [(1 + a enc k ) ⊙ hn,k + b enc k ] , ∆cxn,k = MLPout (1 + a dec k ) ⊙ zn + b dec k
Figure 4 of the paper
Figure 4. Qualitative real-robot rollouts on Organize Books and Pack the Ball. Numbered keyframes show task progression through precise placement and sequential manipulation
Figure 5 of the paper
Figure 5. Action-conditioned scene-flow predictions. Top: observed RGB images. Middle: ini￾tial point clouds. Bottom: predicted endpoint point clouds and scene trajectories (color gradient), conditioned on recorded gripper trajectories (purple). Translucent robot meshes indicate the corre￾sponding poses. diction helps learn representations beneficial to action prediction, consistent with findings in other WAMs (Yuan et al., 2026b). 5 ACTION-CONDITIONED WORLD MODELING Evaluation setup. We evaluate action-conditioned scene prediction on held-out RoboTwin roll￾outs using matched initial…
Figure 6 of the paper
Figure 6. Representative observations and aligned geometry from pretraining data. Robot panels show RGB, depth where available, and scene points with robot overlays. VITRA shows egocentric RGB and hand landmarks. Dataset-specific coordinate conventions. Robot data use a base or shared embodiment frame. Where specified by the bimanual adapter, the origin is the arm-base midpoint, with orientation inher￾ited from the shared root or torso. W1 training coordinates are centered at the median end-effector trajectory midpoint. For VITRA, source camera poses transform hand positions into the current…
Figure 7 of the paper
Figure 7. Additional scene-flow visualizations (1/4): insertion, extraction, bimanual lifting, and
Figure 8 of the paper
Figure 8. Additional scene-flow visualizations (2/4): bowl and mug placement, articulated motion,
Figure 9 of the paper
Figure 9. Additional scene-flow visualizations (3/4): book, bottle, block, and bowl manipulation
Figure 10 of the paper
Figure 10. Additional scene-flow visualizations (4/4): object placement, drawer opening and clos
Figure 11 of the paper
Figure 11. shows predictions on DROID and RH20T conditioned on ground-truth gripper trajectories. Despite receiving scene-flow supervision only from simulation, UNITAS captures action-conditioned scene changes in real environments outside this supervision distribution. These qualitative results suggest that the learned interaction dynamics can transfer from simulation to real scenes. Observation RGB Initial point cloud Predicted midpoint + flows Predicted endpoint + flows Lower the orange object into the metal pan. Lift the grasped item toward the upper cabinet. Lift the patterned cloth…

Figures are the authors', reproduced from the paper.

References (82)
  1. [1] World Models , author =. arXiv preprint arXiv:1803.10122 , year = on PithPith's reviewarXiv
  2. [2] Dream to Control: Learning Behaviors by Latent Imagination , author =. arXiv preprint arXiv:1912.01603 , year = on PithPith's reviewarXiv
  3. [3] Mastering Diverse Domains through World Models , author =. arXiv preprint arXiv:2301.04104 , year = on PithPith's reviewarXiv
  4. [4] Learning Interactive Real-World Simulators , author =. arXiv preprint arXiv:2310.06114 , year = on PithPith's reviewarXiv
  5. [5] Zhen, Haoyu and Qiu, Xiaowen and Chen, Peihao and Yang, Jincheng and Yan, Xin and Du, Yilun and Hong, Yining and Gan, Chuang , journal =. 2024 , url = on Pith
  6. [6] Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations , author =. arXiv preprint arXiv:2412.14803 , year = on PithPith's reviewarXiv
  7. [7] Brohan, Anthony and Brown, Noah and Carbajal, Justice and Chebotar, Yevgen and Chen, Xi and Choromanski, Krzysztof and Ding, Tianli and Driess, Danny and Dubey, Avinava and Finn, Chelsea and Florence, Pete and Fu, Chuyuan and Arenas, Montse Gonzalez and Gopalakrishnan, Keertha... on Pith
  8. [8] Kim, Moo Jin and Pertsch, Karl and Karamcheti, Siddharth and Xiao, Ted and Balakrishna, Ashwin and Nair, Suraj and Rafailov, Rafael and Foster, Ethan and Lam, Grace and Sanketi, Pannag and Vuong, Quan and Kollar, Thomas and Burchfiel, Benjamin and Tedrake, Russ and Sadigh, Dor... on Pith
  9. [9] Black, Kevin and Brown, Noah and Driess, Danny and Esmail, Adnan and Equi, Michael and Finn, Chelsea and Fusai, Niccolo and Groom, Lachy and Hausman, Karol and Ichter, Brian and Jakubczak, Szymon and Jones, Tim and Ke, Liyiming and Levine, Sergey and Li-Bell, Adrian and Mothuk... on Pith
  10. [10] arXiv preprint arXiv:2504.16054 , year =. doi:10.48550/arXiv.2504.16054 , url = on PithPith's reviewarXivdoi:10.48550/arxiv.2504.16054
  11. [11] Unified Video Action Model , author =. arXiv preprint arXiv:2503.00200 , year = on PithPith's reviewarXiv
  12. [12] Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets , author =. arXiv preprint arXiv:2504.02792 , year = on PithPith's reviewarXiv
  13. [13] Zhen, Haoyu and Sun, Qiao and Zhang, Hongxin and Li, Junyan and Zhou, Siyuan and Du, Yilun and Gan, Chuang , journal =. 2025 , url = on Pith
  14. [14] Cen, Jun and Yu, Chaohui and Yuan, Hangjie and Jiang, Yuming and Huang, Siteng and Guo, Jiayan and Li, Xin and Song, Yibing and Luo, Hao and Wang, Fan and Zhao, Deli and Chen, Hao , journal =. 2025 , url = on Pith
  15. [15] Huang, Wenlong and Chao, Yu-Wei and Mousavian, Arsalan and Liu, Ming-Yu and Fox, Dieter and Mo, Kaichun and Fei-Fei, Li , journal =. 2026 , url = on Pith
  16. [16] World Action Models are Zero-shot Policies , author =. arXiv preprint arXiv:2602.15922 , year =. doi:10.48550/arXiv.2602.15922 , url = on PithPith's reviewarXivdoi:10.48550/arxiv.2602.15922
  17. [17] Bi, Hongzhe and Tan, Hengkai and Xie, Shenghao and Wang, Zeyuan and Huang, Shuhe and Liu, Haitian and Zhao, Ruowen and Feng, Yao and Xiang, Chendong and Rong, Yinze and Zhao, Hongyan and Liu, Hanyu and Su, Zhizhong and Ma, Lei and Su, Hang and Zhu, Jun , journal =. 2025 , doi = on Pith
  18. [18] Yuan, Tianyuan and Dong, Zibin and Liu, Yicheng and Zhao, Hang , journal =. 2026 , doi = on Pith
  19. [19] Sun, Jingwen and Zhang, Wenyao and Qi, Zekun and Ren, Shaojie and Liu, Zezhi and Zhu, Hanxin and Sun, Guangzhong and Jin, Xin and Chen, Zhibo , journal =. 2026 , doi = on Pith
  20. [20] Chen, Jialei and Wang, Kai and Chen, Kang and Chen, Shuaihang and Gao, Feng and Tang, Wenhao and Li, Zhiyuan and Liu, Weilin and Yao, Zhuyu and Li, Boxun and Xu, Yuanbo and Yu, Chao , journal =. 2026 , doi = on Pith
show 62 more
  1. [21] Guo, Jun and Li, Qiwei and Li, Peiyan and Chen, Zilong and Sun, Nan and Su, Yifei and Wang, Heyun and Zhang, Yuan and Li, Xinghang and Liu, Huaping , journal =. Unified. 2026 , url = on Pith
  2. [22] Zhou, Pengfei and Chen, Shengcong and Chen, Di and Wang, Jiaxu and Jin, Rongjun and Zhu, Bingwen and Pan, Yike and Gu, Songen and Wang, Kuanning and Nan, Shufeng and others , journal =. 2026 , url = on Pith
  3. [23] Zhao, Haoyu and Zhao, Xingyue and Huang, Siteng and Li, Xin and Zhao, Deli and Li, Zhongyu , journal =. 2026 , url = on Pith
  4. [24] Zhang, Zhanguang and Li, Zhiyuan and Rahmati, Behnam and Yang, Rui Heng and Ma, Yintao and Rasouli, Amir and Pakdamansavoji, Sajjad and Wu, Yangzheng and Zhang, Lingfeng and Cao, Tongtong and Wen, Feng and Wang, Xinyu and Quan, Xingyue and Zhang, Yingxue , journal =. Do World ... on Pith
  5. [25] Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation , author =. Proceedings of the 6th Conference on Robot Learning , series =. 2023 , publisher = on Pith
  6. [26] Ze, Yanjie and Zhang, Gu and Zhang, Kangning and Hu, Chenyuan and Wang, Muhan and Xu, Huazhe , booktitle =. 2024 , doi = on Pith
  7. [27] Bharadhwaj, Homanga and Mottaghi, Roozbeh and Gupta, Abhinav and Tulsiani, Shubham , journal =. 2024 , doi = on Pith
  8. [28] Flow as the Cross-domain Manipulation Interface , author =. Proceedings of The 8th Conference on Robot Learning , year = on Pith
  9. [29] General Flow as Foundation Affordance for Scalable Robot Learning , author =. arXiv preprint arXiv:2401.11439 , year =. doi:10.48550/arXiv.2401.11439 , url = on PithPith's reviewarXivdoi:10.48550/arxiv.2401.11439
  10. [30] Gripper Keypose and Object Pointflow as Interfaces for Bimanual Robotic Manipulation , author =. arXiv preprint arXiv:2504.17784 , year =. doi:10.48550/arXiv.2504.17784 , url = on PithPith's reviewarXivdoi:10.48550/arxiv.2504.17784
  11. [31] Zhi, Hongyan and Chen, Peihao and Zhou, Siyuan and Dong, Yubo and Wu, Quanxi and Han, Lei and Tan, Mingkui , journal =. 2025 , doi = on Pith
  12. [32] 3D Flow Diffusion Policy: Visuomotor Policy Learning via Generating Flow in 3D Space , author =. arXiv preprint arXiv:2509.18676 , year =. doi:10.48550/arXiv.2509.18676 , url = on Pithdoi:10.48550/arxiv.2509.18676
  13. [33] Li, Hongyu and Sun, Lingfeng and Hu, Yafei and Ta, Duy and Barry, Jennifer and Konidaris, George and Fu, Jiahui , journal =. 2025 , doi = on Pith
  14. [34] Zhou, Han and Cao, Jinjin and Ma, Liyuan and Fang, Xueji and Qi, Guo-jun , journal =. 2026 , doi = on Pith
  15. [35] Lee, Seungjae and Jung, Yoonkyo and Chun, Inkook and Lee, Yao-Chih and Cai, Zikui and Huang, Hongjia and Talreja, Aayush and Dao, Tan Dat and Liang, Yongyuan and Huang, Jia-Bin and Huang, Furong , journal =. 2025 , doi = on Pith
  16. [36] Dharmarajan, Karthik and Huang, Wenlong and Wu, Jiajun and Fei-Fei, Li and Zhang, Ruohan , journal =. 2025 , doi = on Pith
  17. [37] Soraki, Rustin and Bharadhwaj, Homanga and Farhadi, Ali and Mottaghi, Roozbeh , journal =. 2026 , doi = on Pith
  18. [38] Wang, Xinkai and Wang, Chenyi and Xu, Yifu and Ye, Mingzhe and Zhang, Fucheng and Tian, Jialin and Zhan, Xinyu and Zhu, Lifeng and Lu, Cewu and Yang, Lixin , journal =. 2026 , doi = on Pith
  19. [39] Lin, Bokai and Xu, Yifu and Zhan, Xinyu and Fang, Hongjie and Tian, Jialin and Zhang, Fu-Cheng and Li, Yong-Lu and Lu, Cewu and Yang, Lixin , journal =. 2026 , doi = on Pith
  20. [40] Zhang, Jianing and Zheng, Chenhao and Yang, Yajun and Argus, Max and Soraki, Rustin and Han, Winson and Anderson, Taira and Li, Chun-Liang and Liu, Shuo and Duan, Jiafei and Ren, Zhongzheng and Zhang, Jieyu and Krishna, Ranjay , journal =. 2026 , doi = on Pith
  21. [41] Qu, Delin and Song, Haoming and Chen, Qizhi and Wang, Dong and Yao, Yuanqi and Ye, Xinyi and Ding, Yan and Wang, Zhigang and Gu, Jiayuan and Zhao, Bin and Li, Xuelong , journal =. 2025 , doi = on Pith
  22. [42] Zheng, Jinliang and Li, Jianxiong and Wang, Zhihao and Liu, Dongxiu and Kang, Xirui and Feng, Yuchun and Zheng, Yinan and Zou, Jiayin and Chen, Yilun and Zeng, Jia and Zhang, Ya-Qin and Pang, Jiangmiao and Liu, Jingjing and Wang, Tai and Zhan, Xianyuan , journal =. 2025 , doi = on Pith
  23. [43] Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots , author =. arXiv preprint arXiv:2402.10329 , year =. doi:10.48550/arXiv.2402.10329 , url = on PithPith's reviewarXivdoi:10.48550/arxiv.2402.10329
  24. [44] Yuan, Haoqi and Liang, Zhixuan and Chen, Anzhe and Wang, Ye and Li, Haoyang and Lin, Pei and Huang, Yiyang and Lei, Zixing and Zhang, Tong and Zhang, Jiazhao and Zhang, Jie and Fan, Jingyang and Zhou, Gengze and Peng, Qihang and Lv, Chenxu and Chen, Xiaoyue and Yang, An and Hu... on Pith
  25. [45] Bai, Yongjie and Wang, Hanting and Dai, Mingtong and Zhong, Qijun and Liu, Yang and Lin, Liang , journal =. 2026 , doi = on Pith
  26. [46] arXiv preprint arXiv:2607.15330 , year =. doi:10.48550/arXiv.2607.15330 , url = on PithPith's reviewarXivdoi:10.48550/arxiv.2607.15330
  27. [47] arXiv preprint arXiv:2503.06669 , year = on PithPith's reviewarXiv
  28. [48] Chen, Tianxing and Chen, Zanxin and Chen, Baijun and Cai, Zijian and Liu, Yibin and Li, Zixuan and Liang, Qiwei and Lin, Xianliang and Ge, Yiheng and Gu, Zhenyu and others , journal = on Pith
  29. [49] Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos , author =. arXiv preprint arXiv:2510.21571 , year = on PitharXiv
  30. [50] Kim, Moo Jin and Gao, Yihuai and Lin, Tsung-Yi and Lin, Yen-Chen and Ge, Yunhao and Lam, Grace and Liang, Percy and Song, Shuran and Liu, Ming-Yu and Finn, Chelsea and Gu, Jinwei , year =. 2601.16163 , archivePrefix = on PithPith's reviewarXiv
  31. [51] Guo, Yanjiang and Shi, Lucy Xiaoyang and Chen, Jianyu and others , year =. 2510.10125 , archivePrefix = on PithPith's reviewarXiv
  32. [52] Causal World Modeling for Robot Control , author =. arXiv preprint arXiv:2601.21998 , year = on PithPith's reviewarXiv
  33. [53] Zhang, Qihang and Li, Lin and Zhang, Luyao and Yang, Shuai and Luo, Yiming and Li, Shuaiting and Wang, Ruilin and Wang, Junke and Shao, Jiahao and Xu, Gangwei and Zhou, Jiaming and Shen, Yishu and Jin, Yudong and Xu, Fangyi and Ma, Shuailei and Liao, Jiaqi and Lu, Guanxing and... on PithPith's reviewarXiv
  34. [54] Yang, Yandan and Zeng, Shuang and Lin, Tong and others , year =. 2602.11236 , archivePrefix = on PithPith's reviewarXiv
  35. [55] Liao, Yue and Zhou, Pengfei and Huang, Siyuan and Yang, Donglin and Chen, Shengcong and Jiang, Yuxin and Hu, Yue and Cai, Jingbin and Liu, Si and Luo, Jianlan and Chen, Liliang and Yan, Shuicheng and Yao, Maoqing and Ren, Guanghui , year =. 2508.05635 , archivePrefix = on PithPith's reviewarXiv
  36. [56] Liang, Weixin and Yu, Lili and Luo, Liang and Iyer, Srinivasan and Dong, Ning and Zhou, Chunting and Ghosh, Gargi and Lewis, Mike and Yih, Wen-tau and Zettlemoyer, Luke and Lin, Xi Victoria , journal =. 2024 , url = on Pith
  37. [57] Liu, Bo and Zhu, Yifeng and Gao, Chongkai and Feng, Yihao and Liu, Qiang and Zhu, Yuke and Stone, Peter , journal =. 2023 , url = on Pith
  38. [58] Zhang, Shiduo and Xu, Zhe and Liu, Peiju and Yu, Xiaopeng and Li, Yuan and Gao, Qinghui and Fei, Zhaoye and Yin, Zhangyue and Wu, Zuxuan and Jiang, Yu-Gang and Qiu, Xipeng , journal =. 2024 , url = on Pith
  39. [59] Khazatsky, Alexander and Pertsch, Karl and Nair, Suraj and others , journal =. 2024 , url = on Pith
  40. [60] Wu, Kun and Hou, Chengkai and Liu, Jiaming and others , journal =. 2024 , url = on Pith
  41. [61] Fang, Hao-Shu and Fang, Hongjie and Tang, Zhenyu and Liu, Jirong and Wang, Chenxi and Wang, Junbo and Zhu, Haoyi and Lu, Cewu , journal =. 2023 , url = on Pith
  42. [62] Scalable Diffusion Models with Transformers , author =. Proceedings of the IEEE/CVF International Conference on Computer Vision , year = on Pith
  43. [63] Ye, Angen and Wang, Boyuan and Ni, Chaojun and Huang, Guan and Zhao, Guosheng and Li, Hao and Li, Hengtao and Li, Jie and Lv, Jindi and Liu, Jingyu and Cao, Min and Li, Peng and Deng, Qiuping and Mei, Wenjun and Wang, Xiaofeng and Chen, Xinze and Zhou, Xinyu and Wang, Yang and... on Pith
  44. [64] Guo, Xinyu and Xie, Bin and Chai, Wei and Deng, Xianchi and Wang, Tiancai and Wu, Zhengxing and Chen, Xingyu , journal =. 2026 , doi = on Pith
  45. [65] Huang, Jinbang and Hu, Yuanzhao and Li, Zhiyuan and Qi, Ran and Xiao, Yixin and Zhang, Zhanguang and Coates, Mark and Cao, Tongtong and Zhang, Yingxue , journal =. 2026 , doi = on Pith
  46. [66] Xiao, Bin and Wu, Haiping and Xu, Weijian and Dai, Xiyang and Hu, Houdong and Lu, Yumao and Zeng, Michael and Liu, Ce and Yuan, Lu , journal =. 2023 , doi = on Pith
  47. [67] Lin, Haotong and Chen, Sili and Liew, Jun Hao and Chen, Donny Y. and Li, Zhenyu and Shi, Guang and Feng, Jiashi and Kang, Bingyi , journal =. 2025 , url = on Pith
  48. [68] Xiao, Yuxi and Wang, Jianyuan and Xue, Nan and Karaev, Nikita and Makarov, Yuri and Kang, Bingyi and Zhu, Xing and Bao, Hujun and Shen, Yujun and Zhou, Xiaowei , booktitle =. 2025 , url = on Pith
  49. [69] Zhou, Huayi and Gao, Wei and Lu, Dekun and Liu, Ruiji and Zhang, Zhanqi and Zhang, Ziyang and Chen, Jian and Zhou, Wenlve and Xu, Sheng and Li, Shumin and Guo, Kangyi and Xu, Shichen and Huang, Zixin and Su, Yongyi and Jia, Kui , journal =. 2026 , url = on Pith
  50. [70] Diffusion-Based Imaginative Coordination for Bimanual Manipulation , author =. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages =. 2025 , doi = on Pith
  51. [71] Wang, Ruixiang and Liu, Qingming and Deng, Yueci and Liu, Guiliang and Liu, Zhen and Jia, Kui , journal =. 2026 , url = on Pith
  52. [72] Shen, Yichao and Wei, Fangyun and Du, Zhiying and Liang, Yaobo and Lu, Yan and Yang, Jiaolong and Zheng, Nanning and Guo, Baining , journal =. 2025 , url = on Pith
  53. [73] Liu, Songming and Wu, Lingxuan and Li, Bangguo and Tan, Hengkai and Chen, Huayu and Wang, Zhengyi and Xu, Ke and Su, Hang and Zhu, Jun , journal =. 2024 , url = on Pith
  54. [74] Steering Robustness into World Action Models via Mechanistic Interpretability and Optimal Control , author =. arXiv preprint arXiv:2607.14943 , year = on PithPith's reviewarXiv
  55. [75] Xiong, Ziyin and Gkanatsios, Nikolaos and Reuss, Moritz and Fragkiadaki, Katerina , booktitle =. 2026 , url = on Pith
  56. [76] Fei, Senyu and Wang, Siyin and Shi, Junhao and Dai, Zihao and Cai, Jikun and Qian, Pengfang and Ji, Li and He, Xinzhe and Zhang, Shiduo and Fei, Zhaoye and Fu, Jinlan and Gong, Jingjing and Qiu, Xipeng , journal =. 2025 , url = on Pith
  57. [77] Chen, Ronghan and Yang, Yandan and Tang, Zuojin and Huo, Dongjie and Lin, Tong and Wu, Haoning and Liu, Haoyun and Chen, Yuzhi and Zheng, Lulu and Yuan, Botai and Li, Tianlun and Wang, Mingxin and Qi, Dekang and Hu, Bin and Mei, Wei and Xuan, Yuze and Yang, Haolong and Zhu, Ya... on Pith
  58. [78] Embodied Hands: Modeling and Capturing Hands and Bodies Together , author =. ACM Transactions on Graphics , volume =. 2017 , url = on Pith
  59. [79] Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow , author =. arXiv preprint arXiv:2209.03003 , year = on PithPith's reviewarXiv
  60. [80] FAST: Efficient Action Tokenization for Vision-Language-Action Models , author =. arXiv preprint arXiv:2501.09747 , year = on PithPith's reviewarXiv
  61. [81] Wan: Open and advanced large-scale video generative models , author=. arXiv preprint arXiv:2503.20314 , year= on PithPith's reviewarXiv
  62. [82] World simulation with video foundation models for physical ai , author=. arXiv preprint arXiv:2511.00062 , year= on PithPith's reviewarXiv
Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment