Pith.

Robotics · arXiv:2610.02840 v1 · submitted 2 Oct 2026 · 69 references

PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation

Chunghyun Park, Beomjun Kim, Seungcheol Park, Heeseung Kwon, Yashu Shukla, Seunghoon Sim, Jinwoo Shin, Minsu Cho

The paper says

“World action models jointly learn to forecast world dynamics and predict robot actions, such that the learned internal world dynamics guide accurate actions.”

more of the abstract

Existing approaches typically represent the world as RGB frames or latent counterparts while predicting actions as end-effector poses or joint angles, but they often struggle to capture the 3D spatial structure and contact geometry central to dexterous manipulation. We introduce Point World Action Model (PointWAM), a 3D world action model that decomposes the world into a scene (i.e., environment) and hands (i.e., actor), and jointly forecasts both as 3D point trajectories within a shared space-time coordinate frame. This explicit, disentangled representation enables effective pre-training on large-scale human demonstration videos without requiring any task-specific object or keypoint selection. Given a colored point cloud and a language instruction, PointWAM predicts how the scene and hands co-evolve in 3D space over time, then retargets the forecast hand motion to robot actions. Pre-training on human videos improves average DexJoCo success by 56.9 percentage points, and scene-trajectory supervision adds 10.9 points over forecasting the hands alone. With both, PointWAM surpasses the prior state of the art on ten DexJoCo tasks by 11.7 points and outperforms strong VLAs on a real robot.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 5 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Real advance in dexterous manipulation with a genuinely new joint scene-plus-hand 3D trajectory formulation and clean pre-training ablations, but the headline scene-supervision gain conflates accurate fine-tuning targets with unvalidated human-video estimates, and the 2D-vs-3D comparison is encoder-confounded. This line is wrong

2 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 69 references say.

The paper says, at §4.4, Table 3(b); §3.4; App. C.1

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The 10.9-point scene-supervision gain, quoted in the abstract (b5), the contribution list (b25), and the conclusion (b163), is measured by removing scene-trajectory supervision from pre-training and fine-tuning simultaneously ('Removing scene-trajectory supervision from both pre-training and fine-tuning'; App. C.1: 'Every variant ... changes both pre-training and fine-tuning').

more of objection 1

The two stages use scene targets that are not interchangeable: at fine-tuning on DexJoCo the targets come from simulator replay with rigid-part segmentation and are accurate (App. B.2), whereas in pre-training on human videos they come from CoTracker3 2D tracks lifted with VGGT-Omega monocular depth whose metric scale is recovered by assuming the ARKit/VITRA hand annotations are correct (App. B.1, 'Depth' and 'Scene tracks'). As measured, the 64.4 vs 53.5 gap is a whole-system property; it cannot distinguish a contribution from the accurate DexJoCo targets (an auxiliary-task or regularization effect at fine-tuning) from a contribution of the estimated human-video targets, or even rule out that the estimated targets add noise the model must unlearn. The conclusion's inference that the world-modeling benefit is carried by the human-video scene forecasts is therefore untested. A 2x2 ablation (scene supervision on/off per stage), or removal of scene supervision from a single stage at a time, would resolve the attribution; without such an experiment, the paper's abstract-level claim that scene-trajectory supervision accounts for 10.9 points is overstated relative to what the evidence supports.

The paper says, at §4.4, Table 3(b), 2D frame row

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The '2D frame' comparison (60.1 vs 64.4) varies two things at once: the forecast target (DINOv3 image latents versus 3D point trajectories) and the encoder (the DINOv3 image encoder versus the Mosaic3D point encoder). Since Table 3(a) shows a 5.4-point gap between exactly these two encoders on the 3D-point pipeline (59.0 with lifted DINOv3 features versus 64.4 with Mosaic3D), part …

more of objection 2

… or all of the 4.3-point gap in panel (b) may be an encoder effect rather than an effect of the trajectory-target form. The sentence 'Forecasting the 3D trajectories of scene points from 3D features thus outperforms forecasting future image latents from image features' attributes the gap to the target form alone, which the reported cells do not isolate. A comparison that holds the encoder fixed (for example, forecasting the 2D latents with the Mosaic3D point encoder, or forecasting 3D point targets with the DINOv3/PointWorld-style lifted features) would support the stated attribution.

4 minor objections

The paper says, at Abstract; §4.3, Figs 6-7

Pith says: minor objection 1 of 4

The 56.9-point pre-training gain in the abstract uses the full-corpus endpoint (12.1% to 69.0%), while Fig. 6 reports 60.1% for EgoDex alone and Fig. 7 shows 69.0% only after adding VITRA. The numbers are consistent, but the abstract should state that the 56.9-point figure corresponds to the combined EgoDex+VITRA corpus to avoid a mismatch for the reader who first sees Fig. 6.

The paper says, at App. C.2

Pith says: minor objection 2 of 4

Checkpoint selection for the V olt-H multi-task model is done on the first evaluation seed's ten-task average (from the 20k/40k/60k fine-tuning snapshots). This is a mild selection-on-evaluation step; the reported three-seed standard deviations would be cleaner if selection used a held-out seed or a validation split. The paper is transparent about the procedure, so this is a protocol nuance rather than an error.

show 2 more

The paper says, at Eq. (8); App. B.1

Pith says: minor objection 3 of 4

The palm-frame transfer to the ten Allegro keypoints via the fixed robot offsets d^k is a central modeling assumption, and no quantitative validation of the retargeting path (for example, correspondence error between human hand keypoints and the induced robot keypoints, or retargeting success on held-out clips) is reported. The paper states the assumption clearly, but a small validation metric would strengthen the claim that the ten-keypoint representation is the right transfer medium.

The paper says, at Real-robot evaluation, Fig. 5; RoboDojo, App. B.2

Pith says: minor objection 4 of 4

The real-robot comparison uses a single training run and 24 trials per method per task with no significance testing, so the reported margins (e.g., 50.0 versus 16.7/25.0 on square-peg insertion) should be read with that uncertainty in mind; the paper does state the trial counts, which is good.

more of objection 4

Separately, on RoboDojo-Precision the scene head is frozen because scene trajectories cannot be reconstructed (App. B.2), so those results exercise the hand-path and action components but not the scene-supervision mechanism and should not be cited as evidence for the scene-supervision claim.

What would settle it

Run the reference ablation (Volt-L12 pre-trained on EgoDex, fine-tuned on DexJoCo) but remove scene-trajectory supervision from pre-training only, keeping it in fine-tuning: pre-train the backbone and hand head with hand-trajectory supervision alone, then fine-tune with the full L_traj of Eq. 7 as in the reference. Compare DexJoCo SR to the reference 64.4% and to the 53.5% no-scene-supervision row. If SR stays at ~64%, the 10.9-point gain comes from accurate fine-tuning scene targets and the estimated human-video scene supervision is not load-bearing, which would reframe the human-video world-modeling claim.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 5 Oct 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic partly agrees 51 s · 25,684 tokens read, 7,284 written
  3. Editor wrote the verdict above 19 s · 26,810 tokens read, 2,968 written
  4. Referee major revision: 2 major objections, 4 minor 1 min 8 s · 26,028 tokens read, 10,573 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

What's new: PointWAM jointly forecasts scene points and hand keypoints as 3D point trajectories in one shared space-time frame and retargets the hands directly, no separate planner. That is genuinely new relative to EgoWAM (grippers, no fingers) and PointWorld/PointZero (scene only, no actions). Pre-training on 1.15M human-video episodes lifts DexJoCo success from 12.1% to 69.0%, beating GR00T N1.6 by 11.7 points. The ablation supporting that pre-training gain is clean: same pipeline, no-pre-training versus with, internal baselines measured identically, scaling curves with data and model size. The paper is honest about its own caveats and states the checkpoint-selection protocol explicitly.

The stress-test point holds, and it is the sharpest issue. The 10.9-point scene-supervision gain (Table 3b) is ablated by removing scene supervision from pre-training and fine-tuning simultaneously. The two stages use different targets: accurate simulator replay segmentation at fine-tuning, unvalidated CoTracker3/VGGT-Omega estimates at pre-training. So the gain cannot be attributed to the human-video scene trajectories; it could live entirely in the fine-tuning stage as an auxiliary-task effect. Validating the estimates would not settle it. The broader "largely from human videos" conclusion still has support from the clean pre-training ablation, but the scene-forecasting component of that transfer specifically is unpinned. One clean split — remove scene supervision from pre-training only, then from fine-tuning only — would resolve it.

Other soft spots, in proportion. The 2D-versus-3D world-modeling row swaps in a different vision encoder (DINOv3), so "3D beats 2D" is confounded with encoder choice; the paper says this itself, but it still undercuts the claim as stated. Real-robot numbers are 24 trials per method, no error bars — a real result on a hard bimanual insertion, just thin. Checkpoint selection on the first seed's ten-task average is transparent and probably modest in effect. None of these touches the core pre-training claim.

Bottom line: the representation and the human-video transfer are real contributions, the SOTA numbers are plausibly supported, and the abstract's scene-supervision attribution needs the disaggregated ablation before the headline holds together. This deserves a serious referee, with a conditional path to accept.

The strongest objection

The paper's world-modeling claim rests on Table 3(b) (wording in §4.4: 'Removing scene-trajectory supervision from both pre-training and fine-tuning'), which yields the 10.9-point figure quoted in the abstract (b5) and contributions (b25). The two stages use different, not interchangeable, scene targets. In fine-tuning on DexJoCo, scene trajectories come from replaying demonstrations with simulator segmentation, which assigns each point to an object or rigid part and defines its trajectory by rigid motion (§B.2); these targets are accurate. In pre-training on human videos, scene trajectories come from CoTracker3 2D tracks lifted with VGGT-Omega monocular depth, whose metric scale is recovered by assuming the ARKit/VITRA hand annotations are correct (§B.1 'Depth', 'Scene tracks'). The ablation removes the supervision in both stages at once, so it cannot distinguish the contribution of the accurate fine-tuning targets from the contribution of the estimated pre-training targets. Consequently the reader's weakest_assumption — that the load-bearing supervision for the world-modeling gain is the unvalidated human-video estimate — is untested in either direction: the 10.9 points could come entirely from the accurate fine-tuning scene loss (an auxiliary-task or regularization effect on simulator targets), with the estimated human-video scene trajectories contributing nothing, or even posing noise the model must unlearn in fine-tuning. If that were true, the paper's distinctive conclusion (b163) that a shared 3D point-trajectory representation lets robots learn dexterous manipulation 'largely from human videos' would weaken: the model would be a hand-keypoint retargeting policy whose scene forecast matters only on robot data. This is a sharper gap than merely 'the estimates are unvalidated': validating the estimates (the reader's suggested remedy) would not settle whether they are load-bearing, since the gain might live entirely in the accurate fine-tuning stage.

The reader's verdict (CONDITIONAL, high confidence) is well-founded: the headline success numbers are supported by clean within-method ablations and external baselines, and the paper is honest about its limitations. My read agrees with keeping the paper CONDITIONAL, but for a partly different reason than the reader's weakest_assumption. The reader located the risk in the unvalidated quality of the CoTracker3+VGGT-Omega scene estimates. I argue the sharper and more load-bearing gap is attribution: the scene-supervision ablation (Table 3b) removes the supervision from both pre-training and fine-tuning at once, and these stages use different target sources — estimated (human videos) versus accurate simulator replay (DexJoCo, §B.2). Even if the estimates were perfect, the 10.9-point gain could be entirely a fine-tuning auxiliary-loss effect, in which case the paper's claim that human-video scene forecasting powers world modeling is unsupported. Conversely, if a pre-training-only ablation reproduces the gain, the estimates become load-bearing and must be validated before the claim is trusted. Neither the reader's framing nor the paper's current experiments distinguish these. The concrete pre-training-only ablation is a single, inexpensive run that settles the question. Secondary concerns (the 2D-vs-3D ablation confounded with encoder choice in Table 3b, checkpoint selection on the first seed's test average in C.2, and the 24-trial real-robot evaluation without error bars) are all real but moderate; they would move the paper from clean to strong but do not touch the core success-rate result, whereas the scene-gain attribution determines whether the paper is a world-action model claim at all. I take no issue with the honesty of the write-up: the paper states its own limitations (b14, b164) and the non-decomposition is a design matter, not a misrepresentation. Conditional acceptance is the right call, with a concrete ablation addressing attribution as the path to accept.

The referee's report

If the reported results hold, PointWAM makes a strong empirical case that explicit 3D point-trajectory world modeling, pre-trained on human egocentric video, can outperform robot-pre-trained VLAs on dexterous manipulation both in simulation and on a small real-robot evaluation. The paper deserves credit for unusually transparent experimental hygiene: the pre-training gain (Fig. 6) and the encoder comparison (Table 3a) hold the rest of the pipeline fixed, training settings are tabulated (Table 4), baseline protocols are specified in detail (App. C.5), and acknowledged limitations (within-clip point correspondence, point-spacing resolution) are stated. The load-bearing gap is the attribution of the scene-supervision benefit: the single ablation removes scene supervision from both training stages at once, so the reported 10.9-point gain is a whole-system property that does not, by itself, identify the estimated human-video scene trajectories as the source of the gain.

Pith's reading of the paper (a machine's summary, not the authors')

PointWAM claims that a robot can acquire dexterous manipulation largely from human demonstration videos, as long as the world is represented as 3D point trajectories — scene and hands in a single space-time frame — and the model is trained to forecast how those points will move before retargeting the forecast hands into actions. Pre-training on 1.15M human episodes from EgoDex and VITRA raises average DexJoCo success from 12.1% to 69.0%, a 56.9-point gain, and supervising the scene's motion adds another 10.9 points over forecasting the hands alone. With both, PointWAM beats the prior best multi-task success rate on DexJoCo by 11.7 points and outperforms robot-pre-trained VLAs on a real robot, including a task with a table-colored object that color-based baselines miss. If correct, the claim would show that explicit 3D scene forecasting from human video is a stronger recipe for dexterous control than image-latent world modeling or vision-language-action policies trained on robot data.

What carries the argument. The load-bearing object is the joint 3D point-trajectory forecast in a shared space-time frame. Scene points and hand keypoints are tokenized onto a common 1 cm voxel grid, aggregated into patches of $K_v^3$ voxels, and processed by a transformer encoder with axial 3D rotary position embeddings; the decoder then emits per-point displacement forecasts for scene and hands from the same features. A frozen Mosaic3D encoder supplies text-aligned 3D point features, and a separate transformer decoder — the action retargeter — turns predicted hand trajectories plus robot state into an action chunk of end-effector poses and finger joints. The mechanism does three jobs at once: it forces the backbone to learn how scene and hand points co-evolve, it lets human videos (which have no robot actions) supply full supervision, and it removes any task-specific object or keypoint selection, since every observed scene point is forecast.

Core claim. PointWAM claims that decomposing the world into a scene and hands, and forecasting both as 3D point trajectories in one shared space-time coordinate frame, is a representation that transfers dexterous skill from human video to robot control. Given a colored point cloud $P_t$ and a language instruction, the model jointly predicts the future displacement of every observed scene point and of ten hand keypoints per hand, $u^{(i)}_{t, au} = p^{(i)}_{t+\tau} - p^{(i)}_t$ and $v^{(k)}_{t, au} = q^{(k)}_{t+\tau} - q^{(k)}_t$, then a retargeter maps the forecast hand trajectories plus robot state into an action chunk. Because human and robot demonstrations collapse to the same trajectory format, one backbone is pre-trained on 1.15M human episodes and fine-tuned on robot demos. The reported numbers: the pre-training data raises DexJoCo success from 12.1% to 69.0%, scene-trajectory supervision adds 10.9 points over forecasting hands alone, and the full model beats the strongest baseline, GR00T N1.6, by 11.7 points and beats robot-pre-trained VLAs on a real robot, with the largest margin on an out-of-domain object whose color resembles the table.

The premise it rests on. The gain credited to scene forecasting (10.9 points) rests on scene trajectories that are never validated: CoTracker3 2D tracks lifted to 3D with VGGT-Ω monocular depth, whose metric scale is recovered by assuming the ARKit and VITRA hand annotations are accurate, so systematic error in those estimates would shrink or erase the world-modeling contribution.

If the paper is right (Pith's inferences, not the authors')

Large-scale human video becomes a viable pre-training corpus for dexterous robotics, reducing reliance on robot demonstrations that are expensive to collect, since only a small robot fine-tuning set (959 DexJoCo demos) is needed after pre-training.

Forecasting the future scene in 3D points beats forecasting it as image latents for dexterous control: the 3D variant gains 10.9 points over hands-only forecasting, while the 2D-latent variant gains 6.6 and still trails the 3D model by 4.3 points, suggesting the gain is specific to 3D trajectory supervision, not world modeling per se.

The point-trajectory representation is not tied to hands: the forecaster transfers to parallel-jaw grippers on RoboDojo-Precision, where human-hand pre-training reaches 8.8 average score and 4.8% success against near-zero scores for video-pretrained Fast-WAM and PointACT, and beats robot-gripper-pretrained GR00T N1.6.

Performance scales with both human-demonstration volume (60.1% with EgoDex alone rising to 69.0% with the full 1.15M-episode corpus) and trainable backbone capacity (52.3% at 39M parameters rising to 69.0% at 655M), so further gains may follow from more human data and larger models.

Under full randomization of camera pose, lighting, and texture, the single-task margin over the strongest baseline widens from 5.3 to 22.2 points, indicating that forecasting 3D scene dynamics is more robust to visual variation than RGB-based methods.

The 3D encoder (Mosaic3D) outperforms lifted image features (DINOv3) by 5.4 points, indicating that encoding 3D structure in the features themselves, rather than only in their positions, matters for dexterous manipulation.

My inference: if the scene-forecasting gain is real, it suggests contact geometry — not appearance — is what dexterous policies most need to anticipate; a natural extension the paper leaves implicit is feeding the forecast scene contact points into the retargeter, which could reduce grasp slippage in tight insertion tasks where the paper's failures cluster.

My inference: the strongest risk to the world-modeling claim is supervision quality — the scene trajectories in human video are estimates (CoTracker3 2D tracks lifted by VGGT-Ω monocular depth scaled to hand annotations), so the scene-forecasting gain could be partly learning the estimator's bias rather than true dynamics; this is testable, because DexJoCo's simulator provides known rigid-body geo

My inference: forecasting every scene point without keypoint selection suggests the approach could generalize to other manipulation modalities where contact points are not known in advance, such as legged manipulation or tool use, where no human annotator can pre-specify the relevant scene points.

My inference: the real-robot evaluation is small (24 trials per task), so the most decision-relevant number to reproduce independently is the out-of-domain banana result (58.3% versus 16.7% and 25.0%), because it claims geometry beats color when appearance cues are ambiguous.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

World action models jointly learn to forecast world dynamics and predict robot actions, such that the learned internal world dynamics guide accurate actions. Existing approaches typically represent the world as RGB frames or latent counterparts while predicting actions as end-effector poses or joint angles, but they often struggle to capture the 3D spatial structure and contact geometry central to dexterous manipulation. We introduce Point World Action Model (PointWAM), a 3D world action model that decomposes the world into a scene (i.e., environment) and hands (i.e., actor), and jointly forecasts both as 3D point trajectories within a shared space-time coordinate frame. This explicit, disentangled representation enables effective pre-training on large-scale human demonstration videos without requiring any task-specific object or keypoint selection. Given a colored point cloud and a language instruction, PointWAM predicts how the scene and hands co-evolve in 3D space over time, then retargets the forecast hand motion to robot actions. Pre-training on human videos improves average DexJoCo success by 56.9 percentage points, and scene-trajectory supervision adds 10.9 points over forecasting the hands alone. With both, PointWAM surpasses the prior state of the art on ten DexJoCo tasks by 11.7 points and outperforms strong VLAs on a real robot.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-05

The authors' main claim, as the classifier picked it out 1.00

“With both, PointWAM surpasses the prior state of the art on ten DexJoCo tasks by 11.7 points and outperforms strong VLAs on a real robot.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.10 · states limitation 0.08 · human participants 0.33 · controlled comparison 0.90 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
World action models jointly learn to forecast world dynamics and predict robot actions, such that the learned internal world dynamics guide accurate actions. background 0.94 0.18 0.12 asserted 1.00
Existing approaches typically represent the world as RGB frames or latent counterparts while predicting actions as end-effector poses or joint angles, but they often struggle to capture the 3D spatial structure and contact geometry central to dexterous manipulation. gap 0.74 0.06 0.03 asserted 0.87
We introduce Point World Action Model (PointWAM), a 3D world action model that decomposes the world into a scene (i.e., environment) and hands (i.e., actor), and jointly forecasts both as 3D point trajectories within a shared space-time coordinate frame. contribution 0.62 0.91 0.08 asserted 1.00
This explicit, disentangled representation enables effective pre-training on large-scale human demonstration videos without requiring any task-specific object or keypoint selection. method 0.49 0.73 0.05 asserted 0.98
Given a colored point cloud and a language instruction, PointWAM predicts how the scene and hands co-evolve in 3D space over time, then retargets the forecast hand motion to robot actions. method 0.99 0.76 0.06 asserted 1.00
Pre-training on human videos improves average DexJoCo success by 56.9 percentage points, and scene-trajectory supervision adds 10.9 points over forecasting the hands alone. result 1.00 0.95 0.16 asserted 1.00
With both, PointWAM surpasses the prior state of the art on ten DexJoCo tasks by 11.7 points and outperforms strong VLAs on a real robot. result 1.00 0.95 0.67 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.92

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-05: 1 request, 5,811 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (12)
Figure 1 of the paper
Figure 1. PointWAM forecasts the scene and hands as 3D point trajectories. PointWAM pre￾trains on human demonstration videos and fine-tunes on robot demonstrations, both expressed as 3D point trajectories (left; depth below RGB). Given a language instruction, its scene and hand forecasts share one space-time coordinate frame (right), and it retargets the hand trajectories to robot actions. 1 INTRODUCTION Dexterous robotic manipulation requires multiple fingers to precisely coordinate their 3D motion relative to object surfaces (Salisbury & Craig, 1982; Okamura et al., 2000), enabling robots…
Figure 2 of the paper
Figure 2. PointWAM architecture. The 3D trajectory forecaster (left) encodes scene points, hand keypoints, and a language instruction, and jointly forecasts scene u and hand v trajectories in a shared space-time coordinate frame. The action retargeter (right) maps v and the robot state st to an action chunk At. MLP denotes a multilayer perceptron, and marks frozen modules. parallel-jaw grippers, a 6-DoF pose and a gripper state per hand, which lacks individual finger mo￾tions. Pri4R (Kim et al., 2026c) likewise targets grippers, adding a training-only 3D point-trajectory head to a VLA…
Figure 3 of the paper
Figure 3. Predicted 3D point trajectories. Scene u and hand v trajectories that PointWAM predicts on unseen DexJoCo layouts; scene trajectories are subsampled for visibility. over the ten tasks, with its standard deviation over seeds. Appendix C.2 details the protocol, and Appendix D reports the results of the single-task policies. RoboDojo-Precision. Although PointWAM is designed for dexterous hands, its point-trajectory representation is not tied to one embodiment. We therefore test whether it transfers to parallel￾jaw grippers, using the Precision tasks of RoboDojo (Chen et al., 2026b),…
Figure 4 of the paper
Figure 4. Qualitative comparison. Rollouts from one initial state per task, each until success (green border) or failure (red border) at its own tend. PointWAM succeeds on both; GR00T N1.6 times out. includes large-scale real and simulated robot demonstrations. Fast-WAM (Yuan et al., 2026) repre￾sents world action models; it builds on a video generation backbone and learns robot actions with video-prediction supervision. PointACT (Chen et al., 2026a) represents policies that use depth, adding pre-trained 3D point features to the action expert of a vision-language policy. Only Fast￾WAM…
Figure 5 of the paper
Figure 5. Real-robot evaluation. (a) The OpenArm platform. (b) The two tasks, one single-arm and one bimanual, with the success rate and score of each method. GR00T N1.6 Fast-WAM PointACT PointWAM 0 20 40 60 80 DexJoCo SR (%) 57.3 56.6 52.9 54.3 33.7 33.9 12.1 60.1 without with
Figure 6 of the paper
Figure 6. Pre-training gain. Average SR (%) with and without EgoDex pre-training. 0 324 736 1148 # episodes (K) 0 20 40 60 80 DexJoCo SR (%) 12.1 60.1 64.9 69.0
Figure 9 of the paper
Figure 9. Successful PointWAM rollouts on DexJoCo. One rollout per task from the multi-task policy, the first success among evaluation seeds 0–7, shown at four points up to its own tend. 22
Figure 10 of the paper
Figure 10. Failed PointWAM rollouts on DexJoCo. One failed rollout per task from the multi-task policy, the first failure among the same rollouts as
Figure 11 of the paper
Figure 11. PointWAM rollouts on RoboDojo-Precision. One rollout per task, shown at four points up to its own tend: a successful one (green border) for the five tasks PointWAM solves at least once, and a failed one (red border) for the other three tasks. 24
Figure 12 of the paper
Figure 12. Real-robot rollouts. GR00T N1.6, π0.5, and PointWAM on the two tasks, each shown at four points up to its own tend with a green border for success and a red border for failure. Banana pick-and-place shows one trial each with the in-domain banana and the out-of-domain unseen box. 25

Showing 10 of the paper's 12 figures. Figures are the authors', reproduced from the paper.

References (69)
  1. [1] Shan An, Ziyu Meng, Chao Tang, et al. Dexterous manipulation through imitation learning: A survey. IEEE Transactions on Automation Science and Engineering (T-ASE), 23, 2026 on Pith
  2. [2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv, 2016 on Pith
  3. [3] Shuai Bai, Keqin Chen, Xuejing Liu, et al. Qwen2.5-VL technical report. arXiv, 2025 on Pith
  4. [4] Chen Bao, Helin Xu, Yuzhe Qin, and Xiaolong Wang. DexArt : Benchmarking generalizable dexterous manipulation with articulated objects. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023 on Pith
  5. [5] Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2Act : Predicting point tracks from internet videos enables generalizable robot manipulation. In European Conference on Computer Vision (ECCV), 2024 on Pith
  6. [6] Hongzhe Bi, Lingxuan Wu, Tianwei Lin, Hengkai Tan, Zhizhong Su, Hang Su, and Jun Zhu. H-RDT : Human manipulation enhanced bimanual robotic manipulation. In AAAI Conference on Artificial Intelligence (AAAI), 2026 on Pith
  7. [7] Antonio Bicchi. Hands for dexterous manipulation and robust grasping: A difficult road toward simplicity. IEEE Transactions on Robotics and Automation (T-RA), 16 0 (6), 2000 on Pith
  8. [8] Kevin Black, Noah Brown, James Darpinian, et al. _ 0.5 : a vision-language-action model with open-world generalization. In Conference on Robot Learning (CoRL), 2025 on Pith
  9. [9] Shizhe Chen, Paul Pacaud, and Cordelia Schmid. PointACT : Vision-language-action models with multi-scale point-action interaction. In Robotics: Science and Systems (RSS), 2026 a on Pith
  10. [10] Tianxing Chen, Yue Chen, Zixuan Li, et al. RoboDojo : A unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies. arXiv, 2026 b on Pith
  11. [11] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems (RSS), 2023 on Pith
  12. [12] Chris Choy and NVIDIA Research . WarpConvNet : High-performance 3D deep learning library. GitHub repository, 2025 on Pith
  13. [13] Lingwei Dang, Juntong Li, Zonghan Li, Hongwen Zhang, Liang An, Wei Min, Yebin Liu, and Qingyao Wu. HarmoHOI : Harmonizing appearance and 3D motion for multi-view hand-object interaction synthesis. In SIGGRAPH Asia Conference Papers, 2026 on Pith
  14. [14] Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. In Advances in Neural Information Processing Systems (NeurIPS), 2023 on Pith
  15. [15] Bardienus P. Duisterhof, Kaifeng Zhang, Adam Hung, Bowen Wen, Stan Birchfield, Yunzhu Li, Deva Ramanan, and Jeffrey Ichnowski. PointZero : 3D point track completion for learning transferable 3D dynamics. arXiv, 2026 on Pith
  16. [16] Yankai Fu, Qiuxuan Feng, Ning Chen, et al. CordViP : Correspondence-based visuomotor policy for dexterous manipulation in real-world. In Robotics: Science and Systems (RSS), 2025 on Pith
  17. [17] GEAR Team . GR00T N1.6 : An improved open foundation model for generalist humanoid robots. NVIDIA Research, 2025 on Pith
  18. [18] Raktim Gautam Goswami, Amir Bar, David Fan, Tsung-Yen Yang, Gaoyue Zhou, Prashanth Krishnamurthy, Michael Rabbat, Farshad Khorrami, and Yann LeCun. World models for learning dexterous hand-object interactions from human videos. In European Conference on Computer Vision (ECCV), 2026 on Pith
  19. [19] Jun Guo, Qiwei Li, Peiyan Li, Zilong Chen, Nan Sun, Yifei Su, Heyun Wang, Yuan Zhang, Xinghang Li, and Huaping Liu. Unified 4D world action modeling from video priors with asynchronous denoising. arXiv, 2026 on Pith
  20. [20] Siddhant Haldar and Lerrel Pinto. Point policy: Unifying observations and actions with key points for robot manipulation. In Conference on Robot Learning (CoRL), 2025 on Pith
show 49 more
  1. [21] Jisang Han, Seonghu Jeon, Jaewoo Jung, René Zurbrügg, Honggyu An, Tifanny Portela, Marco Hutter, Marc Pollefeys, Seungryong Kim, and Sunghwan Hong. Geometric action model for robot policy learning. arXiv, 2026 on Pith
  2. [22] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units ( GELUs ). arXiv, 2016 on Pith
  3. [23] Ryan Hoque, Peide Huang, David J. Yoon, Mouli Sivapurapu, and Jian Zhang. EgoDex : Learning dexterous manipulation from large-scale egocentric video. In International Conference on Learning Representations (ICLR), 2026 on Pith
  4. [24] Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. In International Conference on Machine Learning (ICML), 2025 on Pith
  5. [25] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision (ECCV), 2016 on Pith
  6. [26] Wenlong Huang, Yu-Wei Chao, Arsalan Mousavian, Ming-Yu Liu, Dieter Fox, Kaichun Mo, and Fei-Fei Li. PointWorld : Scaling 3D world models for in-the-wild robotic manipulation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026 on Pith
  7. [27] Adam Hung, Bardienus Pieter Duisterhof, and Jeffrey Ichnowski. 3PoinTr : From human videos to robot policies with 3D point-track plans. arXiv, 2026 on Pith
  8. [28] Jiahao Jiang, Jianing Zhang, Zhenhan Yin, et al. WSA _1 : a 3D -centric world-spatial-action model for generalizable robot control. arXiv, 2026 on Pith
  9. [29] Nikita Karaev, Yuri Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. CoTracker3 : Simpler and better point tracking by pseudo-labelling real videos. In IEEE/CVF International Conference on Computer Vision (ICCV), 2025 on Pith
  10. [30] Beomjun Kim, Seong Hyeon Park, Seunghoon Sim, Seungjun Moon, Sanghyeok Lee, and Jinwoo Shin. Dexterous point policy: Learning point-based dexterous hand policies from human demonstrations. arXiv, 2026 a on Pith
  11. [31] Byungjun Kim, Taeksoo Kim, Junyoung Lee, and Hanbyul Joo. Dexterous world models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026 b on Pith
  12. [32] Jisoo Kim, Jungbin Cho, Sanghyeok Chu, et al. Pri4R : Learning world dynamics for vision-language-action models with privileged 4D representation. In Conference on Robot Learning (CoRL), 2026 c on Pith
  13. [33] Moo Jin Kim, Yihuai Gao, Tsung-Yi Lin, et al. Cosmos policy: Fine-tuning video models for visuomotor control and planning. In International Conference on Learning Representations (ICLR), 2026 d on Pith
  14. [34] Junha Lee, Chunghyun Park, Jaesung Choe, Yu-Chiang Frank Wang, Jan Kautz, Minsu Cho, and Chris Choy. Mosaic3D : Foundation dataset and model for open-vocabulary 3D segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025 on Pith
  15. [35] Seungjae Lee, Yoonkyo Jung, Jusuk Lee, Jonghun Shin, Amir Hossein Shahidzadeh, Yao-Chih Lee, H. Jin Kim, Jia-Bin Huang, and Furong Huang. _0 : A scalable 3D interaction-trace world model. In Conference on Robot Learning (CoRL), 2026 on Pith
  16. [36] Baoyu Li, Xinchen Yin, Mengying Lin, Yixin Zhang, and Danfei Xu. EgoWAM : World action models beyond pixels with in-the-wild egocentric human data. In Conference on Robot Learning (CoRL), 2026 a on Pith
  17. [37] Qixiu Li, Yu Deng, Yaobo Liang, et al. Scalable vision-language-action model pretraining for robotic manipulation with real-life human activity videos. In IEEE International Conference on Robotics and Automation (ICRA), 2026 b on Pith
  18. [38] Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song. Unified video action model. In Robotics: Science and Systems (RSS), 2025 a on Pith
  19. [39] Xianhang Li, Haoqin Tu, Mude Hui, et al. What if we recaption billions of web images with LLaMA -3? In International Conference on Machine Learning (ICML), 2025 b on Pith
  20. [40] Yue Liao, Pengfei Zhou, Siyuan Huang, et al. Genie envisioner: A unified world foundation platform for robotic manipulation. In International Conference on Learning Representations (ICLR), 2026 on Pith
  21. [41] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019 on Pith
  22. [42] Jiangran Lyu, Ziming Li, Xuesong Shi, Chaoyi Xu, Yizhou Wang, and He Wang. DyWA : Dynamics-adaptive world action model for generalizable non-prehensile manipulation. In IEEE/CVF International Conference on Computer Vision (ICCV), 2025 on Pith
  23. [43] NVIDIA , Johan Bjorck, Fernando Castañeda, et al. GR00T N1 : An open foundation model for generalist humanoid robots. arXiv, 2025 on Pith
  24. [44] NVIDIA , Aditi, Niket Agarwal, et al. Cosmos 3: Omnimodal world models for physical AI . arXiv, 2026 on Pith
  25. [45] Allison M. Okamura, Niels Smaby, and Mark R. Cutkosky. An overview of dexterous manipulation. In IEEE International Conference on Robotics and Automation (ICRA), 2000 on Pith
  26. [46] Yuzhe Qin, Binghao Huang, Zhao-Heng Yin, Hao Su, and Xiaolong Wang. DexPoint : Generalizable point cloud reinforcement learning for sim-to-real dexterous manipulation. In Conference on Robot Learning (CoRL), 2022 a on Pith
  27. [47] Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. DexMV : Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision (ECCV), 2022 b on Pith
  28. [48] Delin Qu, Haoming Song, Qizhi Chen, et al. SpatialVLA : Exploring spatial representations for visual-language-action model. In Robotics: Science and Systems (RSS), 2025 on Pith
  29. [49] J. Kenneth Salisbury and John J. Craig. Articulated hands: Force control and kinematic issues. The International Journal of Robotics Research (IJRR), 1 0 (1), 1982 on Pith
  30. [50] Oriane Sim \'e oni, Huy V. Vo, Maximilian Seitzer, et al. DINOv3 . Transactions on Machine Learning Research (TMLR), 2026 on Pith
  31. [51] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer : Enhanced transformer with rotary position embedding. Neurocomputing, 568, 2024 on Pith
  32. [52] Team Wan , Ang Wang, Baole Ai, et al. Wan: Open and advanced large-scale video generative models. arXiv, 2025 on Pith
  33. [53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017 on Pith
  34. [54] Hanwen Wang, Weizhi Zhao, Xiangyu Wang, et al. DexJoCo : A benchmark and toolkit for task-oriented dexterous manipulation on MuJoCo . arXiv, 2026 a on Pith
  35. [55] Jianyuan Wang, Minghao Chen, Shangzhan Zhang, Nikita Karaev, Johannes Sch \"o nberger, Patrick Labatut, Piotr Bojanowski, David Novotny, Andrea Vedaldi, and Christian Rupprecht. VGGT- . In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026 b on Pith
  36. [56] Bowen Wen, Shaurya Dewan, and Stan Birchfield. Fast- FoundationStereo : Real-time zero-shot stereo matching. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026 on Pith
  37. [57] Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. In Robotics: Science and Systems (RSS), 2024 on Pith
  38. [58] Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point Transformer V3 : Simpler, faster, stronger. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024 on Pith
  39. [59] Lishan Yang, Wenxuan Song, Xi Wang, et al. 4D-WAM : Infusing spatiotemporal awareness into world action models through trajectory fields. arXiv, 2026 on Pith
  40. [60] Kadir Yilmaz, Adrian Kruse, Tristan H \"o fer, Daan de Geus, and Bastian Leibe. Volume Transformer : Revisiting vanilla transformers for 3D scene understanding. In European Conference on Computer Vision (ECCV), 2026 on Pith
  41. [61] Chengbo Yuan, Chuan Wen, Tong Zhang, and Yang Gao. General flow as foundation affordance for scalable robot learning. In Conference on Robot Learning (CoRL), 2024 on Pith
  42. [62] Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-WAM : Do world action models need test-time future imagination? arXiv, 2026 on Pith
  43. [63] Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3D diffusion policy: Generalizable visuomotor policy learning via simple 3D representations. In Robotics: Science and Systems (RSS), 2024 on Pith
  44. [64] Huajian Zeng, Lingyun Chen, Jiaqi Yang, Yuantai Zhang, Fan Shi, Peidong Liu, and Xingxing Zuo. FlowHOI : Flow-based semantics-grounded generation of hand-object interactions for dexterous robot manipulation. In Advances in Neural Information Processing Systems (NeurIPS), 2026 on Pith
  45. [65] Jianing Zhang, Chenhao Zheng, Yajun Yang, et al. MolmoMotion : Forecasting point trajectories in 3D with language instruction. arXiv, 2026 on Pith
  46. [66] Yujia Zhang, Xiaoyang Wu, Yixing Lao, Chengyao Wang, Zhuotao Tian, Naiyan Wang, and Hengshuang Zhao. Concerto: Joint 2D - 3D self-supervised learning emerges spatial representations. In Advances in Neural Information Processing Systems (NeurIPS), 2025 on Pith
  47. [67] Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Robotics: Science and Systems (RSS), 2023 on Pith
  48. [68] Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3D-VLA : A 3D vision-language-action generative world model. In International Conference on Machine Learning (ICML), 2024 on Pith
  49. [69] Ruijie Zheng, Jing Wang, Scott Reed, et al. FLARE : Robot learning with implicit world modeling. In Conference on Robot Learning (CoRL), 2025 on Pith
Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment