Pith. sign in

REVIEW 2 major objections 6 minor 21 cited by

Unified Vision-Language-Action Model

T0 review · 2 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single token stream lifts robot policies to new state of the art.

desk verdict A clean unified VLA architecture worth knowing about, but the headline benchmark gains are confounded: the post-training corpus includes videos from the exact environments used for evaluation. read the letter →

arxiv 2506.19850 v1 pith:MAJZ2LYI submitted 2025-06-24 cs.CV cs.RO

Yuqi Wang , Xinghang Li , Wenxuan Wang , Junbo Zhang , Yingyan Li , Yuntao Chen , Xinlong Wang , Zhaoxiang Zhang This is my paper · ORCID
classification cs.CVcs.RO
keywords vision-language-actionmodelworlddiscretetokenizationautoregressivesequencemodelingroboticmanipulationlong-horizonplanningvideopretrainingDCTaction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UniVLA proposes that vision, language, and action should not be treated as separate channels that a model merges at the end, but as one discrete token stream modeled by a single autoregressive transformer. The paper argues that previous vision-language-action models, which bolt an action head onto a vision-language model, miss the temporal and causal structure of robot interaction. By quantizing images with a vector-quantized tokenizer and actions with a Discrete Cosine Transform-based tokenizer into the same vocabulary, UniVLA turns perception, reasoning, and control into the same next-token prediction problem. Post-training the model as a video world model—predicting future frames from an instruction and a current observation—then transfers to policy learning and yields state-of-the-art results on CALVIN, LIBERO, and SimplerEnv-Bridge, including a 95.5% average success rate on LIBERO.

What carries the argument

The load-bearing mechanism is the shared discrete token vocabulary with interleaved causal modeling. Visual observations are compressed by a vector-quantized (VQ) image tokenizer with a spatial compression factor of 8; action windows are transformed with the Discrete Cosine Transform and quantized into tokens from a vocabulary of 1024 that replaces the last 1024 language token IDs; special tokens delimit modality boundaries. All tasks—world modeling, grounding, policy learning—are cast as next-token prediction over a sequence like instruction, observation, action, next observation, next action. The world-model objective is the key training signal: given text instruction and current observation tokens, the model predicts future visual tokens, with loss computed only on vision tokens; the resulting dynamics model is then fine-tuned with loss computed only on action tokens.

What would settle it

Remove CALVIN, LIBERO, and BridgeV2 videos from the post-training corpus (or, symmetrically, give a strong baseline the same extra videos) and re-run the LIBERO and CALVIN evaluations. If LIBERO-Long drops from 94.0% back toward the ~69% prior state of the art, or if the baselines close the gap, the central claim of model-design superiority fails. A second check: replace the discrete DCT action tokens with a continuous action head on the same backbone and compare real-robot or benchmark precision to test whether the tokenization preserves enough control fidelity.

Watch

Extended reading notes

Core claim

The central claim is that a unified, native multimodal model can replace the standard vision-language-action design: instead of a vision encoder feeding a language model that emits actions, UniVLA encodes all three modalities as discrete tokens in a shared vocabulary and trains one autoregressive transformer to predict the next token in an interleaved causal sequence. Actions become frequency-domain tokens via the FAST DCT encoder; images become vector-quantized tokens; text stays as tokens. The same model can then be trained for perception grounding, video world modeling, and policy learning, with the loss computed only on the tokens of the target modality. The paper reports that this design achieves new state-of-the-art results on CALVIN (4.63 and 4.41 average task lengths), LIBERO (95.5% average success, with LIBERO-Long rising from 69.0% to 94.0%), and SimplerEnv-Bridge (69.8% overall vs. 42.7% previous best), and that world-model post-training is what enables fast, data-efficient policy adaptation.

Load-bearing premise

The comparison assumes fairness of training data: the 622K-video post-training corpus includes videos from CALVIN, LIBERO, and BridgeV2, the same environments used for evaluation, so the large reported gains could come from target-domain video exposure rather than from the unified token design or world-model objective.

Editorial extensions

If this is right

  • A single architecture now covers visual prediction, spatial grounding, and action generation, so robot learning and world modeling share one training signal and one set of model weights.
  • World-model post-training makes downstream policy learning data-efficient: with 10% of CALVIN fine-tuning data UniVLA reaches an average task length of 3.19, and by 2,000 training iterations it already scores 4.21.
  • Long-horizon and out-of-distribution tasks benefit most from the world-model stage, as seen in LIBERO-Long jumping from 69.0% to 94.0% success.
  • The same recipe transfers beyond manipulation: fine-tuning on front-camera driving input reaches competitive NAVSIM scores without lidar or bird's-eye-view representations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the post-training corpus includes videos from the exact benchmark environments (CALVIN, LIBERO, BridgeV2), the cleanest test of the method is a held-out-environment evaluation; until that is run, part of the margin over prior methods may be data exposure rather than the token-unification design.
  • Beyond the paper: discrete DCT action tokens trade precision for simplicity; a head-to-head against continuous action heads (e.g., flow matching) on real robots would show whether the tokenization is a constraint or an advantage at high control frequencies.
  • Beyond the paper: if the world-model pretraining generalizes, the recipe suggests an agenda—pretrain on task-agnostic robot video at scale, then fine-tune on small action datasets per embodiment—which the paper partially demonstrates on ALOHA and driving.
  • Beyond the paper: the framework leaves the door open to reinforcement learning by treating reward or value as additional token types, an extension the paper names as future work but does not test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes UniVLA, an 8.5B-parameter encoder-free model that tokenizes vision (via VQ), language, and action (via FAST DCT) into a shared discrete vocabulary and autoregressively predicts interleaved multimodal sequences. The authors first post-train the model as a text-conditioned video world model on 622K robot videos, then fine-tune on policy learning across CALVIN, LIBERO, and SimplerEnv-Bridge. They report state-of-the-art numbers (e.g., 95.5% average on LIBERO vs. pi0-FAST's 85.5%, 4.41/4.63 average length on CALVIN, 69.8% on SimplerEnv-Bridge) and additional multimodal capabilities, real-robot ALOHA results, and an autonomous driving transfer. The central claim is that the unified discrete-token formulation, combined with world-model post-training, yields superior policy learning.

Significance. If the results survive a decontaminated evaluation, the paper would make a strong contribution: it demonstrates that a single next-token predictor can handle vision, language, and action, and that world-model video post-training can accelerate downstream policy learning. The internal ablations (Table 4) are informative and fairly executed across strategies, and the paper is unusually detailed in reporting dataset composition and hyperparameters. However, the headline SOTA claim is currently not supported because the post-training corpus contains videos from the exact benchmark environments used for evaluation; the empirical contribution therefore needs to be re-established under a controlled protocol.

major comments (2)
  1. [Section 4.2 and Appendix Table 8] The post-training corpus includes 22,966 CALVIN videos, 3,386 LIBERO videos, and 28,083 BridgeV2 videos, which are the exact environments/distributions evaluated in Tables 1-3, including SimplerEnv-Bridge. No baseline in those tables received the same target-domain video exposure, so the reported margins (e.g., LIBERO-Long 69.0% to 94.0% in Table 2; SimplerEnv overall 42.7% to 69.8% in Table 3) conflate the world-model objective with privileged access to evaluation environments. The ablation in Table 4 varies only the post-training strategy on this same corpus, and the 'no post-train' row (LIBERO 48.5, CALVIN 1.46) lacks the video exposure entirely, so it cannot separate the two factors. The authors should re-run the main benchmarks after removing all target-domain videos from the post-training corpus, or provide baselines trained with the same additional target-domain videos; Table 4 should be repeated under both conditions.
  2. [Appendix B (Real-Robot Experiments)] The abstract and Section 4.6 claim broad applicability on real-world ALOHA manipulation, but Appendix B reports no quantitative results: no success rates, no per-task evaluation protocol, and no comparison, only a table of collected trajectory counts (Table 9). As written, the real-robot claim is unsupported. Please provide per-task success rates with the number of trials, or explicitly relegate this section to a qualitative proof-of-concept.
minor comments (6)
  1. [Table 6a] The rows in Table 6a lack explicit labels; as printed, the rows with checkmarks are ambiguous about which condition (post-train on/off, visual prediction on/off) they represent.
  2. [Appendix A, Table 8] The ManiSkill2 entry lists 30,213 raw videos but 193,273 used videos, which is impossible; additionally, the column sums to approximately 813K, not the stated 622K total. Please correct the numbers.
  3. [Section 4.2 vs. Appendix A] The post-training stage is described as 30K steps with batch size 64 in Section 4.2, but Appendix A says 50k steps; please unify the numbers.
  4. [Abstract] The abstract spells 'Simplenv-Bridge'; the correct name is SimplerEnv-Bridge.
  5. [Section 1, Contributions] The claim 'the first unified vision-language-action model' is too strong given prior interleaved or joint models (e.g., GR-2, UVA); please soften to avoid overclaiming.
  6. [Figure 3] The caption says 'green boxes indicate predicted object detections,' but it is unclear whether these boxes are model outputs or visual annotations; please clarify.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivation-level circularity: the world-model benefit is an empirical result, not forced by construction, though the post-training corpus overlaps the evaluation benchmarks (a fairness concern, not a circularity).

full rationale

UniVLA's claimed chain is empirical: discrete tokenization (FAST/Emu3 design) -> autoregressive next-token training -> world-model post-training -> policy fine-tuning -> measured success. None of these steps defines the final scores in terms of the inputs; Table 4's ablation compares strategies on the same corpus, and the claim that the world model is best is read off observed results, not derived from the loss definition. The citations to FAST and Emu3 are prior/external; the Emu3 citation (co-author Xinlong Wang) supplies only architecture and initialization, and the efficacy claim does not rest on it. The strongest concern is Appendix Table 8: the post-training corpus contains 22,966 CALVIN videos, 3,386 LIBERO videos, and 28,083 BridgeV2 videos, and the same environments appear in Tables 1-3. This means the 'no post-train' row in Table 4 lacks target-domain video exposure, so the world-model delta conflates the objective with data exposure. That is a benchmark-contamination/fairness threat to the SOTA comparisons, and the paper does not acknowledge it as a limitation, but it is not a self-definition, a fitted-parameter-as-prediction, or a self-citation chain: the success rates are empirically measured and no equation maps the video counts to the reported percentages. Under the hard rules, this is therefore scored as no circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central results rest on standard next-token prediction machinery, plus several domain assumptions about the sufficiency of discrete tokens for control. The most important entry is the last axiom: the post-training video collection includes the exact benchmark environments, which is a data-selection choice specific to this paper. No new physical entities are introduced. The free parameters are design choices (chunk size, history, loss weights, normalization) that were tuned on the benchmarks and influence the reported numbers.

free parameters (5)
  • Action chunk size = 10 (CALVIN/LIBERO), 5 (SimplerEnv)
    Number of future actions predicted per step; chosen per benchmark and affects policy accuracy (Section 4.2).
  • History window = 1 past observation (10 frames)
    Selected via ablation Table 6b; best value is 1 history observation, larger windows reduce CALVIN score.
  • Visual prediction loss weight = 0.5 in no-post-training ablation
    Set to balance vision and action losses when jointly training; Appendix A says visual weight 0.5, action weight 1.0.
  • Action normalization percentiles = 1st and 99th percentile
    Applied to relative actions before DCT tokenization; hand-chosen and affects token distribution (Section 4.2).
  • Post-training frame sampling intervals and dataset caps = dataset-specific intervals (1-20) and 100k cap on Kuka
    Data curation choices that shape world-model training; affect the dynamics seen and thus downstream gains (Appendix Table 8).
assumptions (4)
  • standard math The shared-token autoregressive objective with cross-entropy is a sufficient training signal for multimodal policy learning.
    Invoked in Section 3.1; the paper reduces all tasks to next-token prediction without proving this preserves action-control sufficiency.
  • domain assumption VQ image tokens and DCT action tokens retain the information needed for precise manipulation and world modeling.
    Assumed in Section 3.1 and Figure 2; no reconstruction or control-fidelity analysis is provided.
  • domain assumption The interleaved sequence S_v = {L_t, L_v1,...,L_vt} learns the MDP transition P(s_{t+1}|s_t,a_t) via visual token prediction.
    Stated in Section 3.2 Eq. (1); treats language instruction as action and assumes visual prediction captures dynamics.
  • ad hoc to paper The evaluation benchmarks' own videos (CALVIN, LIBERO, BridgeV2) may be used for post-training without invalidating SOTA comparisons.
    Used in Section 4.2 and Appendix Table 8; this assumption is questionable because prior baselines did not have target-domain video pretraining.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unified Vision-Language-Action Model." pith.science (2026). https://pith.science/paper/MAJZ2LYI

@misc{pith2026250619850,
  author       = {Pith},
  title        = {Pith review of: Unified Vision-Language-Action Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MAJZ2LYI}},
  note         = {Machine review of arXiv:2506.19850}
}
read the original abstract

Vision-language-action models (VLAs) have garnered significant attention for their potential in advancing robotic manipulation. However, previous approaches predominantly rely on the general comprehension capabilities of vision-language models (VLMs) to generate action signals, often overlooking the rich temporal and causal structure embedded in visual observations. In this paper, we present UniVLA, a unified and native multimodal VLA model that autoregressively models vision, language, and action signals as discrete token sequences. This formulation enables flexible multimodal tasks learning, particularly from large-scale video data. By incorporating world modeling during post-training, UniVLA captures causal dynamics from videos, facilitating effective transfer to downstream policy learning--especially for long-horizon tasks. Our approach sets new state-of-the-art results across several widely used simulation benchmarks, including CALVIN, LIBERO, and Simplenv-Bridge, significantly surpassing previous methods. For example, UniVLA achieves 95.5% average success rate on LIBERO benchmark, surpassing pi0-FAST's 85.5%. We further demonstrate its broad applicability on real-world ALOHA manipulation and autonomous driving.

Figures

Figures reproduced from arXiv: 2506.19850 by the authors.

Figure 1
Figure 1. We present UniVLA, a unified vision-language-action model. Unlike prior VLA approaches that typically rely on an extra vision encoder to extract image features and generate only action outputs, UniVLA represents vision, language, and action as discrete tokens within a unified autoregressive framework. This unified modeling paradigm enables multi-modal outputs and supports a wide range of tasks—such as text-supervise… view at source ↗
Figure 2
Figure 2. Overview of the UniVLA framework. Our model unifies information from different modalities into a discrete interleaved sequence, which is modeled using an autoregressive Transformer. To enable unified modeling, images are discretized using vector-quantized (VQ) encoders, while actions are transformed into the frequency domain and discretized via Discrete Cosine Transform (DCT) encoding. This causal multimodal sequenc… view at source ↗
Figure 3
Figure 3. Multimodal capabilities of UniVLA. Top: Action outputs for executing long-horizon tasks in the LIBERO benchmark. Bottom: Visual predictions and spatial grounding demonstrating the model’s spatiotemporal understanding. The red box marks the current observation; green boxes indicate predicted object detections. 4.5 Multimodal Capability As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Real-world setup of the AgileX Cobot Magic dual-arm robot. The system is equipped with three RGB cameras for visual observation: one mounted on the left wrist, one on the right wrist, and one positioned above for a high-angle view. B.1 ALOHA Experimental Setup The robo…
Figure 5
Figure 5. Figure 5: Real-world task examples. These include diverse tasks such as wiping a whiteboard, organizing tableware, making a burger, and plugging in a connector. Real-World Task Collection [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 21 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MobileManiBench: Simplifying Model Verification for Mobile Manipulation

    cs.RO 2026-02 conditional novelty 7.0 of 10

    MobileManiBench is a 300K-trajectory, multi-robot, multi-camera simulated benchmark for VLA model training and evaluation in mobile manipulation.

  2. BrainWAM: Action-Space Coordination of Semantic Priors and Predictive Dynamics for Autonomous Driving

    cs.RO 2026-08 conditional novelty 6.0 of 10

    An action-space coordination framework for combining semantic VLA priors with predictive world-model dynamics achieves 89.5 PDMS on NAVSIM v1 and 89.6 EPDMS on NAVSIM v2.

  3. TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models

    cs.RO 2026-08 conditional novelty 6.0 of 10

    A two-timescale RL post-training method that updates the semantic projection layer rarely and the action expert often improves VLA policy success on long-horizon manipulation tasks.

  4. Deltoris: Enabling Real-time VLA Inference in Embodied AI via Bit-level Sparsity and Speculative Inference

    cs.AR 2026-08 conditional novelty 6.0 of 10

    Temporal-aware bit-level sparsity plus batched speculative verification lets a co-designed accelerator run diffusion-based vision-language-action models up to 34x faster than a mobile GPU.

  5. Latency-Tolerant Cloud-Edge Collaborative Vision-Language-Action Models via Emergent Representational Specialization

    cs.RO 2026-08 conditional novelty 6.0 of 10

    A paired-frame training objective makes cloud VLA features delay-tolerant, letting a lightweight edge head keep acting with stale remote advice and fresh local vision.

  6. CoTinyVLA: Chain-of-Thought Distillation for a Sub-Billion-Parameter Vision-Language-Action Model

    cs.AI 2026-07 conditional novelty 6.0 of 10

    CoTinyVLA, a 0.9B vision-language-action model, outperforms 3-7B baselines on all four LIBERO-Plus robustness suites.

  7. Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    LaMem-VLA reconstructs robotic history into dual short-term and long-term latent memory tokens that are woven directly into a VLA model's reasoning sequence to improve long-horizon manipulation.

  8. Learning 4D Geometric Priors for Inference-Efficient World Action Models

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Training-only multi-expert co-training with decayed 4D read-mask attention and action-aware temporal geometric distillation improves WAM manipulation success while keeping the original lightweight inference graph.

  9. ACE-Brain-0.5: A Unified Embodied Foundational Model for Physical Agentic AI

    cs.RO 2026-07 conditional novelty 6.0 of 10

    A single 8B backbone unifies spatial perception, decision making, navigation/manipulation, and progress estimation with SSR+ merging, reporting gains on most spatial benchmarks and competitive action/progress results.

  10. Perfect Demo Makes Poor Teacher: Learning Robust Alignment from Critical Motion Segments

    cs.RO 2026-06 conditional novelty 6.0 of 10

    Fluent expert demonstrations under-supervise the short alignment phase that decides success, and a compact spatio-temporal dynamic feature (STAIR) recovers most of the deliberate-demonstration gain from fluent data alone.

  11. Guide, Think, Act: Interactive Embodied Reasoning in Vision-Language-Action Models

    cs.RO 2026-05 conditional novelty 6.0 of 10

    GTA-VLA lets humans steer a robot policy with spatial cues (points, boxes, traces) that condition the model's visual chain-of-thought, improving OOD robustness and recovering about 20% of failed episodes.

  12. History-Conditioned Spatio-Temporal Visual Token Pruning for Efficient Vision-Language Navigation

    cs.RO 2026-03 conditional novelty 6.0 of 10

    History-conditioned A-MMR pruning of current and past visual tokens beats prior training-free pruners on R2R/RxR at 70–90% drop rates and runs onboard a Unitree Go2.

  13. Neural Implicit Action Fields: From Discrete Waypoints to Continuous Functions for Vision-Language-Action Models

    cs.RO 2026-03 conditional novelty 6.0 of 10

    NIAF turns robot action chunks into a continuous SIREN function modulated by a VLM, enabling analytic velocity/jerk supervision and state-of-the-art CALVIN/LIBERO results.

  14. Mixture of Horizons in Action Chunking

    cs.RO 2025-11 conditional novelty 6.0 of 10

    A gated mixture of multiple action-chunk horizons in a shared full-attention transformer improves VLA manipulation success and enables consensus-based early stopping.

  15. LHM-Humanoid: Long-Horizon Human Motion Control for Continuous Object Transport in Cluttered Scenes

    cs.RO 2025-08 reject novelty 6.0 of 10

    A staged teacher-student pipeline lets a simulated humanoid relocate two objects in sequence without resets, from egocentric RGB and language, beating the single-task baseline on 350 training and 66 unseen layouts.

  16. MinD: Learning A Dual-System World Model for Real-Time Planning and Implicit Risk Analysis

    cs.RO 2025-06 conditional novelty 6.0 of 10

    A dual-system world model shows that a single-step, not fully denoised, video-diffusion latent is enough to condition a real-time robot manipulation policy, reaching 63% on RL-Bench at 11.3 FPS.

  17. LIRA: Local Cross-Layer Information Routing for Vision-Language-Action Decoding

    cs.RO 2026-08 conditional novelty 5.0 of 10

    A local cross-layer routing mechanism for VLA models, feeding each action-decoder block features from a small window of adjacent VLM layers, improves success rates on LIBERO, CALVIN, and LIBERO-Plus.

  18. Native Video-Action Pretraining for Generalizable Robot Control

    cs.RO 2026-07 conditional novelty 5.0 of 10

    A video-action foundation model pretrained natively with a causal diffusion transformer and semantic visual-action tokenizer reports improved few-shot robot manipulation and 225 Hz asynchronous closed-loop control.

  19. When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs

    cs.CV 2026-02 conditional novelty 5.0 of 10

    VLAs fail most counterfactual instructions because vision shortcuts dominate language; the new LIBERO-CF benchmark quantifies this, and CAG, an inference-time action mixer, improves grounding and success.

  20. Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey

    cs.RO 2025-10 conditional novelty 4.0 of 10

    A survey that groups VLA efficiency techniques into four categories: model architecture, perception features, action generation, and training/inference strategies.

  21. ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceiver

    cs.RO 2025-08 unverdicted novelty 4.0 of 10

    Adding a reconstruction target that redraws the object region makes a vision-language-action model focus its attention on the right object and manipulate more precisely.

Reference graph

Works this paper leans on

81 extracted references · 33 canonical work pages · cited by 21 Pith papers

  1. [1]

    Dream to manipulate: Compositional world models empowering robot imitation learning with imagination.arXiv preprint arXiv:2412.14957, 2024

    Leonardo Barcellona, Andrii Zadaianchuk, Davide Allegro, Samuele Papa, Stefano Ghidoni, and Efstratios Gavves. Dream to manipulate: Compositional world models empowering robot imitation learning with imagination.arXiv preprint arXiv:2412.14957, 2024. 4

    arXiv 2024

  2. [2]

    Rt-h: Action hierarchies using language.arXiv preprint arXiv:2403.01823, 2024

    Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. Rt-h: Action hierarchies using language.arXiv preprint arXiv:2403.01823, 2024. 3

    arXiv 2024

  3. [3]

    Paligemma: A versatile 3b vlm for transfer.arXiv preprint arXiv:2407.07726, 2024

    Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer.arXiv preprint arXiv:2407.07726, 2024. 2

    arXiv 2024

  4. [4]

    Track2act: Predicting point tracks from internet videos enables diverse zero-shot robot manipulation

    Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2act: Predicting point tracks from internet videos enables diverse zero-shot robot manipulation. In European Conference on Computer Vision, 2024. 3

    2024

  5. [5]

    Gr00t n1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025

    Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025. 2

    arXiv 2025

  6. [6]

    π0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024. 2, 3, 4

    arXiv

  7. [7]

    Zero-shot robotic manipulation with pretrained image-editing diffusion models.arXiv preprint arXiv:2310.10639, 2023

    Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models.arXiv preprint arXiv:2310.10639, 2023. 3, 7

    arXiv 2023

  8. [8]

    Rt-2: Vision-language- action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818,

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language- action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818,

    arXiv

Show all 81 references
  1. [9]

    Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022. 7, 8, 16

    2022 arXiv

  2. [10]

    Genie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. InForty-first International Conference on Machine Learning, 2024. 3

    2024

  3. [11]

    Gamegen-x: Interactive open-world game video generation.arXiv preprint arXiv:2411.00769, 2024

    Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamegen-x: Interactive open-world game video generation.arXiv preprint arXiv:2411.00769, 2024. 3

    2024 arXiv

  4. [12]

    Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation.arXiv preprint arXiv:2410.06158, 2024

    Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, et al. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation.arXiv preprint arXiv:2410.06158, 2024. 3

    arXiv

  5. [13]

    Berkeley UR5 demonstration dataset

    Lawrence Yunliang Chen, Simeon Adebola, and Ken Goldberg. Berkeley UR5 demonstration dataset. https://sites.google.com/view/berkeley-ur5/home. 16

  6. [14]

    Playfusion: Skill acquisition via diffusion from language-annotated play

    Lili Chen, Shikhar Bahl, and Deepak Pathak. Playfusion: Skill acquisition via diffusion from language-annotated play. InConference on Robot Learning, pages 2012–2029. PMLR, 2023. 16

    2012

  7. [15]

    Vadv2: End-to-end vectorized autonomous driving via probabilistic planning.arXiv preprint arXiv:2402.13243, 2024

    Shaoyu Chen, Bo Jiang, Hao Gao, Bencheng Liao, Qing Xu, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning.arXiv preprint arXiv:2402.13243, 2024. 9

    2024 arXiv

  8. [16]

    Moto: Latent motion token as the bridging language for robot manipulation.arXiv preprint arXiv:2412.04445, 2024

    Yi Chen, Yuying Ge, Yizhuo Li, Yixiao Ge, Mingyu Ding, Ying Shan, and Xihui Liu. Moto: Latent motion token as the bridging language for robot manipulation.arXiv preprint arXiv:2412.04445, 2024. 8 10

    2024

  9. [17]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023. 7

    2023

  10. [18]

    Transfuser: Imitation with transformer-based sensor fusion for autonomous driving.IEEE transactions on pattern analysis and machine intelligence, 45(11):12878–12895, 2022

    Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving.IEEE transactions on pattern analysis and machine intelligence, 45(11):12878–12895, 2022. 9

    2022

  11. [19]

    Navsim: Data-driven non- reactive autonomous vehicle simulation and benchmarking.Advances in Neural Information Processing Systems, 37:28706–28719, 2024

    Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. Navsim: Data-driven non- reactive autonomous vehicle simulation and benchmarking.Advances in Neural Information Processing...

    2024

  12. [20]

    Humanoid-vla: Towards universal humanoid control with visual integration.arXiv preprint arXiv:2502.14795, 2025

    Pengxiang Ding, Jianfei Ma, Xinyang Tong, Binghong Zou, Xinxin Luo, Yiguo Fan, Ting Wang, Hongchao Lu, Panzhong Mo, Jinxin Liu, et al. Humanoid-vla: Towards universal humanoid control with visual integration.arXiv preprint arXiv:2502.14795, 2025. 2

    2025 arXiv

  13. [21]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conf...

    2021

  14. [22]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. InInternational Conference on Machine Learning, pages 8469–

  15. [23]

    Video language planning.ICLR, 2024

    Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B Tenenbaum, et al. Video language planning.ICLR, 2024. 4

    2024

  16. [24]

    Learning universal policies via text-guided video generation.Advances in neural information processing systems, 36:9156–9172, 2023

    Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation.Advances in neural information processing systems, 36:9156–9172, 2023. 3

    2023

  17. [25]

    Deep visual foresight for planning robot motion

    Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In2017 IEEE international conference on robotics and automation (ICRA), pages 2786–2793. IEEE,

  18. [26]

    Vista: A generalizable driving world model with high fidelity and versatile controllability.arXiv preprint arXiv:2405.17398, 2024

    Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability.arXiv preprint arXiv:2405.17398, 2024. 3

    2024 arXiv

  19. [27]

    Adaworld: Learning adaptable world models with latent actions.arXiv preprint arXiv:2503.18938, 2025

    Shenyuan Gao, Siyuan Zhou, Yilun Du, Jun Zhang, and Chuang Gan. Adaworld: Learning adaptable world models with latent actions.arXiv preprint arXiv:2503.18938, 2025. 4, 8

    2025 arXiv

  20. [28]

    something something

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. InPro...

    2017

  21. [29]

    Maniskill2: A unified benchmark for generalizable manipulation skills

    Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskill2: A unified benchmark for generalizable manipulation skills. InInternational Conferen...

    2023

  22. [30]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 2

    2025 arXiv

  23. [31]

    Prediction with action: Visual policy learning via joint denoising process

    Yanjiang Guo, Yucheng Hu, Jianke Zhang, Yen-Jen Wang, Xiaoyu Chen, Chaochao Lu, and Jianyu Chen. Prediction with action: Visual policy learning via joint denoising process. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 3 11

    2024

  24. [32]

    World models.arXiv preprint arXiv:1803.10122, 2018

    David Ha and Jürgen Schmidhuber. World models.arXiv preprint arXiv:1803.10122, 2018. 3

    2018 arXiv

  25. [33]

    Dream to control: Learning behaviors by latent imagination.arXiv preprint arXiv:1912.01603, 2019

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination.arXiv preprint arXiv:1912.01603, 2019. 3

    1912 arXiv

  26. [34]

    Learning latent dynamics for planning from pixels

    Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. InInternational conference on machine learning, pages 2555–2565. PMLR, 2019. 4

    2019

  27. [35]

    Gaia-1: A generative world model for autonomous driving

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080, 2023. 3

    2023 arXiv

  28. [36]

    Planning-oriented autonomous driving

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17853–17862, 2023. 9

    2023

  29. [37]

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al.\pi_{0.5}: a vision- language-action model with open-world generalization.arXiv preprint arXiv:2504.16054, 2025. 3

    2025 arXiv

  30. [38]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024. 2

    2024 arXiv

  31. [39]

    Scalable deep reinforcement learning for vision-based robotic manipulation

    Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. InConference on robot learning, pag...

    2018

  32. [40]

    Droid: A large-scale in-the-wild robot manipulation dataset.arXiv preprint arXiv:2403.12945, 2024

    Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset.arXiv preprint arXiv:2403.12...

    2024 arXiv

  33. [41]

    Fine-tuning vision-language-action models: Optimizing speed and success.arXiv preprint arXiv:2502.19645, 2025

    Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success.arXiv preprint arXiv:2502.19645, 2025. 3

    2025 arXiv

  34. [42]

    Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024. 2, 3, 4, 7, 16

    2024 arXiv

  35. [43]

    A path towards autonomous machine intelligence version 0.9

    Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1–62, 2022. 3

    2022

  36. [44]

    Gr-mg: Leveraging partially-annotated data via multi-modal goal-conditioned policy.IEEE Robotics and Automation Letters, 2025

    Peiyan Li, Hongtao Wu, Yan Huang, Chilam Cheang, Liang Wang, and Tao Kong. Gr-mg: Leveraging partially-annotated data via multi-modal goal-conditioned policy.IEEE Robotics and Automation Letters, 2025. 3

    2025

  37. [45]

    Unified video action model.arXiv preprint arXiv:2503.00200, 2025

    Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song. Unified video action model.arXiv preprint arXiv:2503.00200, 2025. 4

    2025 arXiv

  38. [46]

    Towards generalist robot policies: What matters in building vision-language-action models.arXiv preprint arXiv:2412.14058, 2024

    Xinghang Li, Peiyan Li, Minghuan Liu, Dong Wang, Jirong Liu, Bingyi Kang, Xiao Ma, Tao Kong, Hanbo Zhang, and Huaping Liu. Towards generalist robot policies: What matters in building vision-language-action models.arXiv preprint arXiv:2412.14058, 2024. 7, 8, 16

    2024 arXiv

  39. [47]

    Vision-language foundation models as effective robot imitators

    Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators. InICLR, 2024. 7

    2024

  40. [48]

    Evaluating real-world robot manipulation policies in simulation.arXiv preprint arXiv:2405.05941, 2024

    Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, et al. Evaluating real-world robot manipulation policies in simulation.arXiv preprint arXiv:2405.05941, 2024. 2, 6 12

    2024 arXiv

  41. [49]

    Libero: Benchmarking knowledge transfer for lifelong robot learning.Advances in Neural Information Processing Systems, 36:44776–44791, 2023

    Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning.Advances in Neural Information Processing Systems, 36:44776–44791, 2023. 2, 6, 16

    2023

  42. [50]

    Hybridvla: Collaborative diffusion and autoregression in a unified vision-language-action model.arXiv preprint arXiv:2503.10631,

    Jiaming Liu, Hao Chen, Pengju An, Zhuoyang Liu, Renrui Zhang, Chenyang Gu, Xiaoqi Li, Ziyu Guo, Sixiang Chen, Mengzhen Liu, et al. Hybridvla: Collaborative diffusion and autoregression in a unified vision-language-action model.arXiv preprint arXiv:2503.10631,

    arXiv

  43. [51]

    Fmb: a functional manipulation benchmark for generalizable robotic learning

    Jianlan Luo, Charles Xu, Fangchen Liu, Liam Tan, Zipeng Lin, Jeffrey Wu, Pieter Abbeel, and Sergey Levine. Fmb: a functional manipulation benchmark for generalizable robotic learning. The International Journal of Robotics Research, page 02783649241276017, 2023. 16

    2023

  44. [52]

    Language conditioned imitation learning over unstructured data.arXiv preprint arXiv:2005.07648, 2020

    Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data.arXiv preprint arXiv:2005.07648, 2020. 7

    2005 arXiv

  45. [53]

    What matters in language conditioned robotic imitation learning over unstructured data.IEEE Robotics and Automation Letters, 7(4):11205–11212, 2022

    Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data.IEEE Robotics and Automation Letters, 7(4):11205–11212, 2022. 8

    2022

  46. [54]

    Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks.IEEE Robotics and Automation Letters, 7(3):7327–7334, 2022

    Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks.IEEE Robotics and Automation Letters, 7(3):7327–7334, 2022. 2, 5, 16

    2022

  47. [55]

    R3m: A universal visual representation for robot manipulation.arXiv preprint arXiv:2203.12601, 2022

    Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation.arXiv preprint arXiv:2203.12601, 2022. 8

    2022 arXiv

  48. [56]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source general...

  49. [57]

    Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023. 2

    2023 arXiv

  50. [58]

    Fast: Efficient action tokenization for vision-language-action models.arXiv preprint arXiv:2501.09747, 2025

    Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-language-action models.arXiv preprint arXiv:2501.09747, 2025. 3, 4, 5, 6, 7

    2025 arXiv

  51. [59]

    Spatialvla: Exploring spatial representations for visual-language-action model.arXiv preprint arXiv:2501.15830, 2025

    Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, JiaYuan Gu, Bin Zhao, Dong Wang, et al. Spatialvla: Exploring spatial representations for visual-language-action model.arXiv preprint arXiv:2501.15830, 2025. 3, 7

    2025 arXiv

  52. [60]

    Latent plans for task-agnostic offline reinforcement learning

    Erick Rosete-Beas, Oier Mees, Gabriel Kalweit, Joschka Boedecker, and Wolfram Burgard. Latent plans for task-agnostic offline reinforcement learning. InConference on Robot Learning, pages 1838–1849. PMLR, 2023. 16

    2023

  53. [61]

    Mutex: Learning unified policies from multimodal task specifications.arXiv preprint arXiv:2309.14320, 2023

    Rutav Shah, Roberto Martín-Martín, and Yuke Zhu. Mutex: Learning unified policies from multimodal task specifications.arXiv preprint arXiv:2309.14320, 2023. 16

    2023 arXiv

  54. [62]

    Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213, 2024

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213, 2024. 7

    2024 arXiv

  55. [63]

    Predictive inverse dynamics models are scalable learners for robotic manipulation.arXiv preprint arXiv:2412.15109, 2024

    Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong, and Jiangmiao Pang. Predictive inverse dynamics models are scalable learners for robotic manipulation.arXiv preprint arXiv:2412.15109, 2024. 7

    2024 arXiv

  56. [64]

    Open x-embodiment: Robotic learning datasets and rt-x models

    Quan Vuong, Sergey Levine, Homer Rich Walke, Karl Pertsch, Anikait Singh, Ria Doshi, Charles Xu, Jianlan Luo, Liam Tan, Dhruv Shah, et al. Open x-embodiment: Robotic learning datasets and rt-x models. InTowards Generalist Robots: Learning Paradigms for Scalable Skill Acquisiti...

    2023

  57. [65]

    Bridgedata v2: A dataset for robot learning at scale

    Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. InConference on Robot Learning, pages 1723–1736. PMLR, 2023. 16

    2023

  58. [66]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 2

    2024 arXiv

  59. [67]

    Drive- dreamer: Towards real-world-drive world models for autonomous driving

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drive- dreamer: Towards real-world-drive world models for autonomous driving. InEuropean Confer- ence on Computer Vision, pages 55–72. Springer, 2024. 3

    2024

  60. [68]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 4, 6

    2024 arXiv

  61. [69]

    Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving

    Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1...

    2024

  62. [70]

    Unleashing large-scale video generative pre-training for visual robot manipulation

    Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. InThe Twelfth International Conference on Learning Representations,

  63. [71]

    Day- dreamer: World models for physical robot learning

    Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Day- dreamer: World models for physical robot learning. InConference on robot learning, pages 2226–2240. PMLR, 2023. 3, 4

    2023

  64. [72]

    Learning interactive real-world simulators.arXiv preprint arXiv:2310.06114, 1(2):6, 2023

    Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators.arXiv preprint arXiv:2310.06114, 1(2):6, 2023. 3

    2023 arXiv

  65. [73]

    Latent action pretraining from videos.ICLR, 2025

    Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, et al. Latent action pretraining from videos.ICLR, 2025. 3, 8

    2025

  66. [74]

    Up-vla: A unified understanding and prediction model for embodied agent.arXiv preprint arXiv:2501.18867, 2025

    Jianke Zhang, Yanjiang Guo, Yucheng Hu, Xiaoyu Chen, Xiang Zhu, and Jianyu Chen. Up-vla: A unified understanding and prediction model for embodied agent.arXiv preprint arXiv:2501.18867, 2025. 7

    2025 arXiv

  67. [75]

    Cot-vla: Visual chain-of-thought reasoning for vision-language-action models.arXiv preprint arXiv:2503.22020, 2025

    Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, et al. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models.arXiv preprint arXiv:2503.22020, 2025. 6, 7

    2025 arXiv

  68. [76]

    3d-vla: A 3d vision-language-action generative world model

    Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d vision-language-action generative world model. InInternational Conference on Machine Learning, pages 61229–61245. PMLR, 2024. 3

    2024

  69. [77]

    Movq: Modulating quantized vectors for high-fidelity image generation.Advances in Neural Information Processing Systems, 35:23412–23425, 2022

    Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation.Advances in Neural Information Processing Systems, 35:23412–23425, 2022. 4

    2022

  70. [78]

    Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies.arXiv preprint arXiv:2412.10345, 2024

    Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé III, Andrey Kolobov, Furong Huang, and Jianwei Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies.arXiv preprint arXiv:2412.10345, 2024. 3

    2024 arXiv

  71. [79]

    Train offline, test online: A real robot learning benchmark

    Gaoyue Zhou, Victoria Dean, Mohan Kumar Srirama, Aravind Rajeswaran, Jyothish Pari, Kyle Hatch, Aryan Jain, Tianhe Yu, Pieter Abbeel, Lerrel Pinto, et al. Train offline, test online: A real robot learning benchmark. In2023 IEEE International Conference on Robotics and Automati...

    2023

  72. [80]

    Ro- bodreamer: Learning compositional world models for robot imagination.arXiv preprint arXiv:2404.12377, 2024

    Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. Ro- bodreamer: Learning compositional world models for robot imagination.arXiv preprint arXiv:2404.12377, 2024. 3

    2024 arXiv

  73. [81]

    Viola: Imitation learning for vision- based manipulation with object proposal priors

    Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Imitation learning for vision- based manipulation with object proposal priors. InConference on Robot Learning, pages 1199–1210. PMLR, 2023. 16 15 Appendix A Implementation Details Post-training StageWe began by sele...

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.