Pith. sign in

REVIEW 4 major objections 7 minor 71 references

Decoupling video prediction from action prediction, then guiding both with affordance and depth, yields stronger robot policies from fewer demos.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 22:20 UTC pith:VLPOZMR6

load-bearing objection Solid modular VAM recipe with real efficiency evidence; sim story holds, real-world headline is softer than the abstract implies. the 4 major comments →

arxiv 2607.24159 v1 pith:VLPOZMR6 submitted 2026-07-27 cs.RO cs.CV

DeVA: Decoupled Video-Action Model with physical guidance for robot policy learning

Mengqi Zhang , Sahil Khose , Simar Kareer , Yuchen Song , Unnat Jain , Judy Hoffman This is my paper
classification cs.RO cs.CV
keywords Video-Action Modelrobot manipulationvideo generative modelsdecoupled expertsaffordancedepth guidancepolicy learningVision-Language-Action
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Language-conditioned robot policies need to anticipate how a scene will change when the robot moves, but standard vision-language models are pretrained mostly on static images and leave dynamics to scarce robot demos. Video generative models already encode those dynamics by predicting future frames. This paper argues that the useful way to use them is not to force video and actions through one shared backbone, and not to treat the video model as a black-box encoder. Instead it builds two specialized experts: a video expert that imagines future observations, and an action expert that outputs control. Intermediate video features at many layers are passed to the action expert through cross-attention and compact bridge tokens, and the video features are further shaped by affordance and relative-depth supervision so the action stream sees where to interact and how the scene is laid out in depth. On kitchen and tabletop simulation suites and on real bimanual tasks, the design reaches high success with only tens of demonstrations per task, converges faster than a matched unified architecture, and gains measurably from the physical guidance.

Core claim

DeVA shows that robot policy learning from video generative priors works better when video dynamics and action prediction are handled by separate transformer experts linked by multi-level feature transfer and by explicit affordance and depth guidance, rather than by a single shared backbone or by sparse, single-layer video conditioning. Under limited demonstrations this yields higher success, faster optimization, and clear gains from the physical signals on both simulation benchmarks and real bimanual hardware.

What carries the argument

The decoupled video-action architecture: a pretrained video diffusion expert and a separate action diffusion expert, connected by layer-wise cross-attention plus learnable bridge tokens, with intermediate video features also decoded into task-conditioned affordance and relative-depth maps that supervise the video stream and are injected as guidance tokens into the action expert.

Load-bearing premise

The claim depends on intermediate video features, shaped by imperfect affordance and depth labels, actually carrying the right control information across the multi-level interface rather than the gains coming mostly from extra model capacity or training setup.

What would settle it

Train a capacity-matched unified backbone and a decoupled model without affordance/depth guidance on the same filtered demos and budget; if success and convergence gaps on RoboCasa, LIBERO, and the three real bimanual tasks disappear, the central architectural claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Video world-model pretraining can be turned into data-efficient robot policies without forcing video and actions into one shared representation.
  • Multi-layer video features plus compact bridge tokens are a practical interface for transferring predictive dynamics into control heads.
  • Lightweight affordance and depth supervision can steer generative video features toward interaction regions and geometry that matter for manipulation.
  • Under matched optimization budgets, specialized experts can converge faster than unified video-action backbones on the same tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the physical decoders stay useful after freezing in stage two, similar auxiliary geometry or contact heads could regularize other generative backbones used only as frozen feature banks.
  • The real-robot gains with thirty demos per task suggest the same recipe may transfer to other multi-camera bimanual setups where large action pretraining corpora are unavailable.
  • A natural next stress test is whether multi-level transfer still helps when the video expert is trained on much longer horizons or open-world video rather than short task clips.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper presents DeVA, a video-action model for language-conditioned manipulation in which a pretrained video diffusion transformer (Cosmos-Predict2, 2B params) and a separate flow-matching action expert are trained jointly but kept parameter-decoupled, exchanging information through layer-wise cross-attention from eight video layers plus learnable bridge tokens. Intermediate video features are additionally supervised with task-conditioned affordance heatmaps and relative depth via lightweight DPT-style decoders, whose features are injected as guidance tokens into the action expert's cross-attention. The authors report: (i) RoboCasa 72.0% success with only 50 demos/task, exceeding baselines trained on 300–3,000 demos; (ii) 99.0% average on LIBERO and 80.8% on LIBERO-Plus (+11.2 over the best baseline); (iii) faster convergence and much higher success than a matched unified-backbone variant (66–71% vs 34% at ~40–50K steps); (iv) real-world bimanual results on three tasks with an average "success rate" of 74% vs 48% (GR00T-N1.6) and 34% (Cosmos-Policy); and (v) ablations attributing ~6 points on RoboCasa to affordance+depth guidance. The simulation evidence is substantial; the real-world evidence is weaker than the headline numbers suggest, for reasons detailed below.

Significance. If the results hold, DeVA is a useful contribution to the video-action-model line of work: it gives a concrete, ablated answer to how to interface a pretrained video diffusion backbone with an action expert (layer-wise cross-attention plus bridge tokens, multi-step denoising features), and the RoboCasa data-efficiency result (72.0% at 50 demos/task, with 3,600 rollouts over 3 seeds) and the LIBERO-Plus robustness margin (+11.2 over the strongest baseline) are practically meaningful. The component ablations (Fig. 8), coupled-vs-decoupled convergence comparison under matched budget (Fig. 7(b)), and detailed appendix (parameter accounting in Table 2, per-dataset normalization in Table 4, denoising-step and interaction-mechanism ablations in Tables 7–9) are the kind of evidence that lets others build on the work. The promised release of code, configs, and checkpoints, and the documented real-world evaluation criteria (App. C.2), are genuine reproducibility strengths. The contribution is incremental relative to concurrent dual-DiT VAMs [9,10] in architecture, but the physical-guidance mechanism and the systematic efficiency analysis differentiate it.

major comments (4)
  1. [§4.4, Fig. 6(b), Table 6, Appendix C.2] Fig. 6(b) and Table 6 vs. Appendix C.2: the quantity reported as 'Success Rate' for real-world deployment is not a success rate. Appendix C.2 defines a stage-completion partial-credit score in [0,1] (e.g., Lift Pot awards 0.25/0.5/0.75 for incomplete episodes; Handover Marker awards 0.5 when one gripper lifts but the transfer fails). With 10 trials per task, a binary success rate must be a multiple of 0.1, yet Table 6 reports 0.88, 0.45, 0.23, 0.68 — values only possible under partial credit. This is load-bearing for the abstract and §4.4 claim 'DeVA achieves an average success rate of 74%, compared with 48%/34%': partial credit inflates all methods, and it does so differentially depending on where credit is granted (e.g., the 0.5 bin for 'one gripper lifts' is generous and may reward failure modes unevenly across methods). The fix is straightforward but mandatory: relabel the metric as
  2. [§4.4, Fig. 6(b), Table 6] Table 6 and Fig. 6(b): the real-world comparison in the main text omits the strongest baseline. Table 6 (appendix) reports π0.5 at 0.67 average vs. DeVA's 0.74, yet Fig. 6(b) and §4.4 compare only against GR00T-N1.6 (0.48) and Cosmos-Policy (0.34), and the abstract-level real-world claim rests on that comparison. With n=10 trials per task per method, the per-task standard error under the partial-credit metric is roughly 8–12 points, so the 0.74 vs. 0.67 gap over π0.5 is well within noise, and even the 0.74 vs. 0.48 gap over GR00T-N1.6 is only about 2σ on the per-task level. π0.5 must be included in the main-text figure and discussion (including, per §C.3, the caveat that π0.5 brings large-scale robot-centric pretraining and a 50-step action horizon vs. 25 for DeVA), and the claims should be tempered to what n=10×3 tasks can support — or the trial count increased. As written, the headline
  3. [§5, Fig. 8(b), Appendix B.3] Fig. 8(b) and Appendix B.3: the claim of 'clear performance gains from physical guidance' is established only in simulation, where affordance targets are oracle MuJoCo contacts/projected end-effector positions (§B.3). In the real-world pipeline the affordance targets are pseudo-labels from an off-the-shelf model [50] used 'out-of-the-box' and depth targets come from [57]; the authors themselves note these contain 'moderate prediction noise.' No real-world ablation isolates the contribution of physical guidance, so it is possible that on hardware the guidance tokens add little or that noisier pseudo-labels erode the sim-measured 6-point gain (66.0→72.0). The paper should either add a real-world ablation (base vs. +guidance on at least one task) or explicitly scope the physical-guidance claim to simulation in the abstract and §7. Given that the guidance module is a named contribution of th
  4. [§4.5, Fig. 7(b)] Fig. 7(b): the 'faster convergence than a unified architecture' claim is supported only over the plotted horizon (~60K steps). The coupled variant is at ~34% at 40–50K steps when the decoupled variants plateau at 66–71%, but the figure does not show whether the coupled model eventually catches up with more steps, i.e., whether decoupling buys asymptotic performance or only optimization speed. This matters because the coupled variant is the paper's own construction (not a published baseline), so the reader cannot rule out that it is simply under-tuned. Please either extend the coupled curve to convergence, or state explicitly that the claim is about matched-budget efficiency, and report the coupled variant's final performance under its own best budget.
minor comments (7)
  1. [§3.3 / Table 5] §3.3 defines Stage-1 loss weights λ_aff and λ_depth, but their values appear only in Appendix Table 5 (both 1.0). State the values in the main text where the objective is introduced.
  2. [Table 1, Fig. 5] Simulation results use 3 seeds × 50 rollouts, but no variance or confidence intervals are reported in Table 1 or Fig. 5. On LIBERO, DeVA's 99.0 vs. DiT4DiT's 98.6 and Cosmos-Policy's 98.5 is almost certainly within seed noise; please report standard errors so that 'highest overall average' claims are calibrated.
  3. [Fig. 5(c)] Fig. 5(c) ('Perturbation') is never referenced or described in the text; the caption mentions 'Robot Init' and 'Camera View' axes but no numbers or discussion are given. Either integrate it into §4.3 or remove it.
  4. [Fig. 3] Fig. 3 contains a label 'Bit Addition' (presumably an addition/fusion op); clarify or correct the terminology.
  5. [Appendix D.3, Table 9] Appendix D.3 reports 30K-step checkpoint results (49.6/53.2/22.4) that are far below the ~55K-step numbers elsewhere (e.g., 66.0 base). Please note the different budget explicitly in the table caption to avoid confusion with main-text numbers.
  6. [General] Formatting/typos: 'DeV A' appears with a spurious space throughout the extracted text (e.g., title, §3, §4.1); Table 1's last row reads 'Ours5072.0' with missing column separation; §4.1 says 'three complementary simulation benchmarks' and the introduction says 'three simulation benchmarks, including RoboCasa, LIBERO, LIBERO-plus' — 'including' suggests the list is non-exhaustive; make it exact.
  7. [Appendix B.1] §B.1: training uses Cosmos-Policy's replay-filtered (success-only) demonstrations for LIBERO and RoboCasa. It would help to state explicitly which baselines in Table 1 and Fig. 5(a) were trained on the same filtered data versus their original data, since filtering changes the effective data budget (1,199 episodes for RoboCasa across 24 tasks).

Circularity Check

0 steps flagged

No significant circularity: empirical architecture and policy results evaluated on external simulators and robots, not algebraic restatements of fitted inputs.

full rationale

DeVA is an empirical robotics/ML paper. Its load-bearing claims are comparative success rates, convergence curves, and ablations on RoboCasa, LIBERO/LIBERO-Plus, and real bimanual hardware. Those metrics are produced by rolling out policies in external simulators or on physical robots under stated protocols; they are not defined by the training losses or by parameters fitted to the same quantities being reported. Video and action objectives (EDM denoising, flow matching), multi-level cross-attention, and affordance/depth auxiliary losses are design choices supervised by simulator contacts, off-the-shelf depth/affordance models, or pseudo-labels—standard external supervision, not self-definition of the evaluation metric. Pretrained Cosmos weights and community-filtered demonstration sets are ordinary transfer/data practice and do not force the reported margins by construction. There is no uniqueness theorem, self-citation chain that substitutes for evidence, or renaming of a known identity as a derived prediction. Concerns about partial-credit real-world scoring, small trial counts, or missing baselines affect evaluation integrity, not circularity of a derivation chain.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 2 invented entities

Load-bearing content is architectural and empirical, not a short formal derivation. The claim rests on standard generative-model training assumptions, the hypothesis that video diffusion intermediates encode control-useful dynamics across layers, and the adequacy of affordance/depth supervision—including pseudo-labels in the real world—plus many optimization and interface hyperparameters chosen for the method to work.

free parameters (5)
  • λ_aff and λ_depth (Stage-1 auxiliary loss weights) = 1.0 / 1.0
    Balance video denoising against affordance/depth heads; set to 1.0 in Appendix without principled derivation from data likelihood.
  • Number and placement of transferred video layers / bridge tokens = 8 layers; 12 tokens × 1024
    Eight uniformly sampled video blocks routed to action cross-attention and 12 learnable bridge tokens (dim 1024) are design choices that define the interface capacity.
  • Learning-rate piecewise schedule and Stage-1/2 step budgets = peak 1e-4; ~55K sim / 40K real
    Peak 1e-4 with warmup/decay breakpoints and ~10K warmup then ~40–55K joint steps are hand-chosen optimization knobs that affect the convergence claims.
  • Affordance Gaussian kernel and positive-class weighting = 3×3 Gaussian σ=1.0
    Contact points are smoothed (3×3, σ=1.0) and BCE is reweighted; these labeling choices shape the physical guidance signal.
  • Noise-weighting w(t)=1/(1+σ_t)^2 for auxiliary losses = w(t)=1/(1+σ_t)^2
    Couples physical supervision strength to diffusion noise level by formula choice, not cross-validated necessity.
axioms (6)
  • domain assumption Pretrained video diffusion backbones encode spatiotemporal dynamics useful for downstream robot control when fine-tuned with future-frame objectives.
    Stated in §1 and motivates initializing from Cosmos-Predict2; standard in VAM literature but not proved here.
  • domain assumption Diffusion features at different layers/denoising stages carry complementary abstractions worth multi-level transfer to actions.
    Invoked in §3.1 citing diffusion feature papers [11,12]; justifies layer-wise cross-attention design.
  • domain assumption Task-conditioned affordance maps and relative depth are sufficiently complete physical signals to shape intermediate video features for manipulation.
    §3.2 defines At(u,v) via end-effector interaction likelihood and adds depth; alternative geometric cues are not compared exhaustively.
  • ad hoc to paper Off-the-shelf affordance and video-depth models provide adequate pseudo-labels for real-world supervision without metric multi-view consistency.
    Appendix B.2–B.3 explicitly accept noise and cross-view inconsistency for real data; load-bearing for real deployment claims.
  • standard math EDM video denoising and action flow-matching are appropriate generative objectives for joint video-action learning.
    Standard training losses carried over from Cosmos/GR00T-style stacks (§3.3, Appendix A).
  • ad hoc to paper Decoupling video and action parameters makes policy optimization more tractable than a shared backbone under matched budgets.
    Central architectural hypothesis tested via coupled vs decoupled curves in Fig. 7(b); assumed mechanism is optimization geometry, not formally characterized.
invented entities (2)
  • DeVA dual-expert Video-Action architecture with bridge-token interface no independent evidence
    purpose: Separate video dynamics modeling from action prediction while enabling structured multi-level information flow.
    Named system combining specialized DiT experts, layer-wise cross-attention, and learnable bridge tokens; evaluated only inside this paper’s experiments.
  • Physical guidance tokens from affordance/depth DPT-style decoders no independent evidence
    purpose: Inject explicit interaction-region and geometry cues into action cross-attention beyond raw video latents.
    Decoder features are projected, concatenated with video keys/values, and treated as a distinct conditioning channel (§3.2, Fig. 3).

pith-pipeline@v1.2.0-grok45-kimik3 · 25876 in / 4065 out tokens · 83334 ms · 2026-07-31T22:20:04.996419+00:00 · methodology

0 comments
read the original abstract

Generalizable robot manipulation requires policies that can anticipate how visual scenes evolve while executing language instructions. While recent Vision-Language-Action models benefit from large-scale pretraining, their predominantly static pretraining objectives provide limited supervision for physical dynamics and temporal causality, leaving control-relevant knowledge to be learned from downstream robot demonstrations. Video generative models offer a promising foundation by encoding rich spatiotemporal priors through future predictions. However, existing Video-Action Models either couple video and action prediction in a shared backbone, making policy adaptation harder to optimize, or under-utilize video information when guiding the action branch. In this work, we introduce DeVA, a Decoupled Video-Action model with specialized video and action experts, multi-level feature transfer, and physically salient guidance. DeVA transfers representations from multiple video layers to the action expert, enabling rich information exchange while making policy learning more tractable. It further supervises intermediate video features and the action stream with physically salient guidance (affordance/depth). Experiments on both simulation benchmarks and real-world deployment demonstrate strong performance with limited data, faster convergence than a unified architecture, and clear performance gains from physical guidance.

Figures

Figures reproduced from arXiv: 2607.24159 by Judy Hoffman, Mengqi Zhang, Sahil Khose, Simar Kareer, Unnat Jain, Yuchen Song.

Figure 1
Figure 1. Figure 1: Overview of DeVA compared with VLA and coupled VAM. VLA directly maps image￾language inputs to actions without explicit dynamics modeling, while coupled video-action models jointly process video and action tokens in a shared representation. DeVA decouples video dynamics modeling from action prediction with physical guidance, enabling efficient robot policy learning. Abstract: Generalizable robot manipulati… view at source ↗
Figure 2
Figure 2. Figure 2: Overview. DeVA decouples video dynamics modeling and action prediction. The action stream attends to layer-wise video features through cross-attention to produce robot action chunks. Multi-Level Feature Interaction. Although separate experts preserve modality-specific capacity, fully isolating them would prevent the action expert from accessing the predictive dynamics learned by the video backbone. Effecti… view at source ↗
Figure 3
Figure 3. Figure 3: Physical guidance module. The module aggregates multi-level video features and decodes them into affordance and depth maps, which are then converted into physically grounded tokens and injected into the action transformer head to support action prediction. expert can therefore attend to both predictive video dynamics and explicit cues about where and how the end effector should act. In this way, physically… view at source ↗
Figure 4
Figure 4. Figure 4: RoboCasa task rollout examples. Success Rate # Demos / Task Avg. GR00T-N1 [22] 300 49.6 DP-VLA [61] 3000 57.3 GR00T-N1 + DreamGen [33] 300 (+10K syn.) 57.6 GR00T-N1 + DUST [62] 300 58.5 π0 [2] 300 62.5 GR00T-N1.5 [22] 300 64.1 FLARE [36] 300 66.4 GR00T-N1.5 + HAMLET [63] 300 66.4 Video-Action Models Video Policy [38] 300 66.0 UVA [37] 50 50.0 UWM [5] 1000 60.8 Cosmos-Policy [6] 50 67.1 Ours 50 72.0 [PITH_… view at source ↗
Figure 5
Figure 5. Figure 5: Simulation benchmark comparisons on LIBERO and LIBERO-Plus. The robustness exam￾ples are on "Robot Init" and "Camera View" axes. Simulation Benchmarks. We evaluate DeVA on three complementary simulation benchmarks covering multitask manipulation, diverse environments, and robustness under controlled shifts: • RoboCasa [13]. A large-scale simulation benchmark featuring realistic kitchen en￾vironments and di… view at source ↗
Figure 6
Figure 6. Figure 6: Real-robot keyframes for three tasks’ completion and quantitative success rates com￾parison with the other two VLA/VAM. 4.4 Real-World Deployment. We evaluate DeVA on three bimanual tasks using the I2RT platform: Handover Marker, Lift Pot, and Pick Up Bottles. For each task, we collect 30 ex￾pert demonstrations and fine-tune DeVA, Cosmos Policy, and GR00T-N1.6 in a multi-task manner and using the same opti… view at source ↗
Figure 7
Figure 7. Figure 7: Comparison over training optimization and data efficiency. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Ablations of the video-prediction archi [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Visualization of physical guidance across representative manipulation examples. Affordance [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Keyframes of the Lift Pot task, including coordinated approach, synchronized grasping, [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Keyframes of the Handover Marker task, highlighting grasping, precise bimanual transfer, [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Keyframes of the Pick Up Diverse Bottles task, including coordinated dual-arm grasping, [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Robust analysis examples along three axes: instance, distraction, and layout. [PITH_FULL_IMAGE:figures/full_fig_p023_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

71 extracted references · 24 linked inside Pith

  1. [1]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, et al. Openvla: An open-source vision-language-action model. In Conference on Robot Learning, pages 2679–2713. PMLR, 2025

    2025

  2. [2]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. π0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Intelligence, K

    P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. π0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Zitkovich, T

    B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. InConference on Robot Learning, pages 2165–2183. PMLR, 2023

    2023

  5. [5]

    C. Zhu, R. Yu, S. Feng, B. Burchfiel, P. Shah, and A. Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets. InProceedings of Robotics: Science and Systems (RSS), 2025

    2025

  6. [6]

    M. J. Kim, Y . Gao, T.-Y . Lin, Y .-C. Lin, Y . Ge, G. Lam, P. Liang, S. Song, M.-Y . Liu, C. Finn, and J. Gu. Cosmos policy: Fine-tuning video models for visuomotor control and planning. In International Conference on Learning Representations (ICLR), 2026

    2026

  7. [7]

    S. Ye, Y . Ge, K. Zheng, S. Gao, S. Yu, G. Kurian, S. Indupuru, Y . L. Tan, C. Zhu, J. Xiang, A. Malik, K. Lee, W. Liang, N. Ranawaka, J. Gu, Y . Xu, G. Wang, F. Hu, A. Narayan, J. Bjorck, J. Wang, G. Kim, D. Niu, R. Zheng, Y . Xie, J. Wu, Q. Wang, R. Julian, D. Xu, Y . Du, Y . Chebotar, S. Reed, J. Kautz, Y . Zhu, L. J. Fan, and J. Jang. World action mod...

  8. [8]

    Y . Hu, Y . Guo, P. Wang, X. Chen, Y .-J. Wang, J. Zhang, K. Sreenath, C. Lu, and J. Chen. Video prediction policy: A generalist robot policy with predictive visual representations. In International Conference on Machine Learning, pages 24328–24346. PMLR, 2025

    2025

  9. [9]

    J. Pai, L. Achenbach, V . Montesinos, B. Forrai, O. Mees, and E. Nava. mimic-video: Video- action models for generalizable robot control beyond vlas.arXiv preprint 2512.15692, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    T. Ma, J. Zheng, Z. Wang, C. Jiang, A. Cui, J. Liang, and S. Yang. Dit4dit: Jointly modeling video dynamics and actions for generalizable robot control.arXiv preprint arXiv:2603.10448, 2026

    arXiv 2026

  11. [11]

    G. Luo, L. Dunlap, D. H. Park, A. Holynski, and T. Darrell. Diffusion hyperfeatures: Search- ing through time and space for semantic correspondence.Advances in Neural Information Processing Systems, 36:47500–47510, 2023

    2023

  12. [12]

    D. Kim, X. Thomas, and D. Ghadiyaram. Revelio: Interpreting and leveraging semantic information in diffusion models. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4659–4669, 2025

    2025

  13. [13]

    Nasiriany, A

    S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y . Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots. InRobotics: Science and Systems (RSS), 2024. 9

    2024

  14. [14]

    B. Liu, Y . Zhu, C. Gao, Y . Feng, Q. Liu, Y . Zhu, and P. Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning.Advances in Neural Information Processing Systems, 36: 44776–44791, 2023

    2023

  15. [15]

    S. Fei, S. Wang, J. Shi, Z. Dai, J. Cai, P. Qian, L. Ji, X. He, S. Zhang, Z. Fei, et al. Libero-plus: In- depth robustness analysis of vision-language-action models.arXiv preprint arXiv:2510.13626, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

    Pith/arXiv arXiv 2022

  17. [17]

    Driess, F

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. InInternational Conference on Machine Learning, pages 8469–8488. PMLR, 2023

    2023

  18. [18]

    Bousmalis, G

    K. Bousmalis, G. Vezzani, D. Rao, C. Devin, A. X. Lee, M. Bauzá, T. Davchev, Y . Zhou, A. Gupta, A. Raju, et al. Robocat: A self-improving generalist agent for robotic manipulation. arXiv preprint arXiv:2306.11706, 2023

    Pith/arXiv arXiv 2023

  19. [19]

    Ghosh, H

    Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y . Tan, L. Y . Chen, P. Sanketi, Q. Vuong, T. Xiao, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy. InProceedings of Robotics: Science and Systems, Delft, Netherlands, 2024

    2024

  20. [20]

    Q. Li, Y . Liang, Z. Wang, L. Luo, X. Chen, M. Liao, F. Wei, Y . Deng, S. Xu, Y . Zhang, et al. Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation.arXiv preprint arXiv:2411.19650, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    S. Liu, L. Wu, B. Li, H. Tan, H. Chen, Z. Wang, K. Xu, H. Su, and J. Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation. InInternational Conference on Learning Representations, volume 2025, pages 29982–30009, 2025

    2025

  22. [22]

    Bjorck, F

    J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y . Fang, D. Fox, F. Hu, S. Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Pertsch, K

    K. Pertsch, K. Stachowicz, B. Ichter, D. Driess, S. Nair, Q. Vuong, O. Mees, C. Finn, and S. Levine. Fast: Efficient action tokenization for vision-language-action models.arXiv preprint arXiv:2501.09747, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    M. J. Kim, C. Finn, and P. Liang. Fine-tuning vision-language-action models: Optimizing speed and success.arXiv preprint arXiv:2502.19645, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    H. Zhen, X. Qiu, P. Chen, J. Yang, X. Yan, Y . Du, Y . Hong, and C. Gan. 3d-vla: A 3d vision- language-action generative world model. InInternational Conference on Machine Learning, pages 61229–61245. PMLR, 2024

    2024

  26. [26]

    D. Qu, H. Song, Q. Chen, Y . Yao, X. Ye, Y . Ding, Z. Wang, J. Gu, B. Zhao, D. Wang, et al. Spatialvla: Exploring spatial representations for visual-language-action model.arXiv preprint arXiv:2501.15830, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Zheng, Y

    R. Zheng, Y . Liang, S. Huang, J. Gao, H. Daumé III, A. Kolobov, F. Huang, and J. Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. InInternational Conference on Learning Representations, volume 2025, pages 54277– 54296, 2025

    2025

  28. [28]

    Q. Bu, Y . Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li. Univla: Learning to act anywhere with task-centric latent actions.arXiv preprint arXiv:2505.06111, 2025. 10

    Pith/arXiv arXiv 2025

  29. [29]

    W. Li, R. Zhang, R. Shao, J. He, and L. Nie. Cogvla: Cognition-aligned vision-language- action model via instruction-driven routing & sparsification.Advances in neural information processing systems, 2025

    2025

  30. [30]

    Y . Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation.Advances in neural information processing systems, 36:9156–9172, 2023

    2023

  31. [31]

    Black, M

    K. Black, M. Nakamoto, P. Atreya, H. Walke, C. Finn, A. Kumar, and S. Levine. Zero- shot robotic manipulation with pre-trained image-editing diffusion models. InInternational Conference on Learning Representations, volume 2024, pages 33431–33452, 2024

    2024

  32. [32]

    Bharadhwaj, D

    H. Bharadhwaj, D. Dwibedi, A. Gupta, S. Tulsiani, C. Doersch, T. Xiao, D. Shah, F. Xia, D. Sadigh, and S. Kirmani. Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation. InConference on Robot Learning, pages 3936–3951. PMLR, 2025

    2025

  33. [33]

    J. Jang, S. Ye, Z. Lin, J. Xiang, J. Bjorck, Y . Fang, F. Hu, S. Huang, K. Kundalia, Y .-C. Lin, et al. Dreamgen: Unlocking generalization in robot learning through video world models. In Conference on Robot Learning, pages 5170–5194. PMLR, 2025

    2025

  34. [34]

    S. Ye, J. Jang, B. Jeon, S. J. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y .-W. Chao, B. Y . Lin, et al. Latent action pretraining from videos. InInternational Conference on Learning Representations, volume 2025, pages 28213–28239, 2025

    2025

  35. [35]

    H. Wu, Y . Jing, C. Cheang, G. Chen, J. Xu, X. Li, M. Liu, H. Li, and T. Kong. Unleashing large- scale video generative pre-training for visual robot manipulation. InInternational Conference on Learning Representations, volume 2024, pages 10641–10662, 2024

    2024

  36. [36]

    Zheng, J

    R. Zheng, J. Wang, S. Reed, J. Bjorck, Y . Fang, F. Hu, J. Jang, K. Kundalia, Z. Lin, L. Magne, et al. Flare: Robot learning with implicit world modeling. InConference on Robot Learning, pages 3952–3971. PMLR, 2025

    2025

  37. [37]

    S. Li, Y . Gao, D. Sadigh, and S. Song. Unified video action model.arXiv preprint arXiv:2503.00200, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    Liang, P

    J. Liang, P. Tokmakov, R. Liu, S. Sudhakar, P. Shah, R. Ambrus, and C. V ondrick. Video generators are robot policies.arXiv preprint arXiv:2508.00795, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    Routray, H

    S. Routray, H. Pan, U. Jain, S. Bahl, and D. Pathak. Vipra: Video prediction for robot actions. arXiv preprint arXiv:2511.07732, 2025

    arXiv 2025

  40. [40]

    T. Yuan, Z. Dong, Y . Liu, and H. Zhao. Fast-wam: Do world action models need test-time future imagination?arXiv preprint arXiv:2603.16666, 2026

    Pith/arXiv arXiv 2026

  41. [41]

    K. Mo, L. J. Guibas, M. Mukadam, A. Gupta, and S. Tulsiani. Where2act: From pixels to actions for articulated 3d objects. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 6813–6823, 2021

    2021

  42. [42]

    S. Deng, X. Xu, C. Wu, K. Chen, and K. Jia. 3d affordancenet: A benchmark for visual object affordance understanding. Inproceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1778–1787, 2021

    2021

  43. [43]

    S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak. Affordances from human videos as a versatile representation for robotics. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13778–13790, 2023

    2023

  44. [44]

    C. Yuan, C. Wen, T. Zhang, and Y . Gao. General flow as foundation affordance for scalable robot learning. InConference on Robot Learning, pages 1541–1566. PMLR, 2025. 11

    2025

  45. [45]

    W. Yuan, J. Duan, V . Blukis, W. Pumacay, R. Krishna, A. Murali, A. Mousavian, and D. Fox. Robopoint: A vision-language model for spatial affordance prediction in robotics. InConference on Robot Learning, pages 4005–4020. PMLR, 2025

    2025

  46. [46]

    Huang, C

    W. Huang, C. Wang, Y . Li, R. Zhang, and L. Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. InConference on Robot Learning, pages 4573–4602. PMLR, 2025

    2025

  47. [47]

    Nasiriany, S

    S. Nasiriany, S. Kirmani, T. Ding, L. Smith, Y . Zhu, D. Driess, D. Sadigh, and T. Xiao. Rt- affordance: Affordances are versatile intermediate representations for robot manipulation. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 8249–8257. IEEE, 2025

    2025

  48. [48]

    K. Rana, J. Abou-Chakra, S. Garg, R. Lee, I. Reid, and N. Suenderhauf. Learning from 10 demos: Generalisable and sample-efficient policy learning with oriented affordance frames. In Conference on Robot Learning, pages 5464–5482. PMLR, 2025

    2025

  49. [49]

    S. Wu, Y . Zhu, Y . Huang, K. Zhu, J. Gu, J. Yu, Y . Shi, and J. Wang. Afforddp: Generalizable diffusion policy with transferable affordance. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 6971–6980, 2025

    2025

  50. [50]

    Y . Tang, W. Huang, Y . Wang, C. Li, R. Yuan, R. Zhang, J. Wu, and L. Fei-Fei. Uad: Un- supervised affordance distillation for generalization in robotic manipulation. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 3822–3831. IEEE, 2025

    2025

  51. [51]

    Shridhar, L

    M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. InConference on Robot Learning, pages 785–799. PMLR, 2023

    2023

  52. [52]

    Goyal, J

    A. Goyal, J. Xu, Y . Guo, V . Blukis, Y .-W. Chao, and D. Fox. Rvt: Robotic view transformer for 3d object manipulation. InConference on Robot Learning, pages 694–710. PMLR, 2023

    2023

  53. [53]

    Gervet, Z

    T. Gervet, Z. Xian, N. Gkanatsios, and K. Fragkiadaki. Act3d: 3d feature field transformers for multi-task robotic manipulation. InConference on Robot Learning, pages 3949–3965. PMLR, 2023

    2023

  54. [54]

    Y . Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. InProceedings of Robotics: Science and Systems (RSS), 2024

    2024

  55. [55]

    Y . Jia, J. Liu, S. Chen, C. Gu, Z. Wang, L. Luo, L. Lee, P. Wang, Z. Wang, R. Zhang, et al. Lift3d foundation policy: Lifting 2d large-scale pretrained models for robust 3d robotic manipulation. arXiv preprint arXiv:2411.18623, 2024

    Pith/arXiv arXiv 2024

  56. [56]

    L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao. Depth anything v2. Advances in Neural Information Processing Systems, 37:21875–21911, 2024

    2024

  57. [57]

    S. Chen, H. Guo, S. Zhu, F. Zhang, Z. Huang, J. Feng, and B. Kang. Video depth anything: Consistent depth estimation for super-long videos. In2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22831–22840. IEEE, 2025

    2025

  58. [58]

    L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. InCVPR, 2024

    2024

  59. [59]

    L. Fan, Z. Xu, C. Cao, W. Zhang, M. Yuan, and J. Chen. Aim: Intent-aware unified world action modeling with spatial value maps.arXiv preprint arXiv:2604.11135, 2026

    Pith/arXiv arXiv 2026

  60. [60]

    Cosmos-predict2: World simulation model for physical ai, 2025

    NVIDIA. Cosmos-predict2: World simulation model for physical ai, 2025. URL https: //github.com/nvidia-cosmos/cosmos-predict2. 12

    2025

  61. [61]

    B. Han, J. Kim, and J. Jang. A dual process vla: Efficient robotic manipulation leveraging vlm. arXiv preprint arXiv:2410.15549, 2024

    Pith/arXiv arXiv 2024

  62. [62]

    J. Won, K. Lee, H. Jang, D. Kim, and J. Shin. Dual-stream diffusion for world-model augmented vision-language-action model.arXiv preprint arXiv:2510.27607, 2025

    Pith/arXiv arXiv 2025

  63. [63]

    M. Koo, D. Choi, T. Kim, K. Lee, C. Kim, Y . Seo, and J. Shin. Hamlet: Switch your vision- language-action model into a history-aware policy.arXiv preprint arXiv:2510.00695, 2025

    Pith/arXiv arXiv 2025

  64. [64]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 44(10-11):1684–1704, 2025

    2025

  65. [65]

    J. Cen, C. Yu, H. Yuan, Y . Jiang, S. Huang, J. Guo, X. Li, Y . Song, H. Luo, F. Wang, et al. Worldvla: Towards autoregressive action world model.arXiv preprint arXiv:2506.21539, 2025

    Pith/arXiv arXiv 2025

  66. [66]

    C.-Y . Hung, Q. Sun, P. Hong, A. Zadeh, C. Li, U. Tan, N. Majumder, S. Poria, et al. Nora: A small open-sourced generalist vision language action model for embodied tasks.arXiv preprint arXiv:2504.19854, 2025

    Pith/arXiv arXiv 2025

  67. [67]

    S. Tan, K. Dou, Y . Zhao, and P. Krähenbühl. Interactive post-training for vision-language-action models.arXiv preprint arXiv:2505.17016, 2025

    Pith/arXiv arXiv 2025

  68. [68]

    Y AM Arm Series.https://doc.i2rt.com/products/yam, 2024

    I2RT Robotics. Y AM Arm Series.https://doc.i2rt.com/products/yam, 2024

    2024

  69. [69]

    Cheng, J

    X. Cheng, J. Li, S. Yang, G. Yang, and X. Wang. Open-television: Teleoperation with immersive active visual feedback. InConference on Robot Learning, pages 2729–2749. PMLR, 2025

    2025

  70. [70]

    J. Xu, S. Liu, A. Vahdat, W. Byeon, X. Wang, and S. De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2955–2966, 2023. 13 Appendix A Implementation Details A.1 Video Expert Architecture In DeV A, we adopt Cosmos-Predict2 [60] as th...

    2023

  71. [2026]

    URLhttps://arxiv.org/abs/2602.15922

    Pith/arXiv arXiv