Pith. sign in

REVIEW 4 major objections 7 minor 11 cited by

Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoning

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single 7B VLM can serve as both slow reasoner and fast robot controller.

desk verdict Solid dual-system VLA with a genuinely new shared-parameter twist; the mechanism itself is confounded by pretraining and modality changes, but the system and ablations still deserve review. read the letter →

arxiv 2506.01953 v1 pith:N474HSTI submitted 2025-06-02 cs.RO

Hao Chen , Jiaming Liu , Chenyang Gu , Zhuoyang Liu , Renrui Zhang , Xiaoqi Li , Xiao He , Yandong Guo
show 3 more authors
Chi-Wing Fu Shanghang Zhang Pheng-Ann Heng
This is my paper · ORCID
classification cs.RO
keywords vision-language-actionmodeldual-systemarchitecturefastandslowsystemsdiffusionpolicyautoregressivereasoningroboticmanipulationhigh-frequencycontrolpointcloudconditioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a robot's fast, reactive action loop does not need a separate lightweight policy model; instead, the final transformer blocks of an intact vision-language model can themselves become the fast executor. If true, this would mean internet-scale pretrained knowledge, which normally lives in the slow reasoning layers, is directly available to the high-frequency control loop, closing the usual gap between smart planning and quick acting. The authors embed System 1 inside System 2 by sharing parameters, give the two systems different input modalities and asynchronous operating frequencies, and train them jointly with both a diffusion action loss and an autoregressive next-token loss. On their reported evaluations, the approach lifts average success by 8 percent in simulation and 11 percent on two real-world dual-arm platforms relative to prior state-of-the-art dual-system methods, while pushing control frequency above 20 Hz at action chunk one and over 117 Hz at chunk eight.

What carries the argument

The load-bearing object is the shared-block split: the final $N$ transformer blocks of a pretrained 7B LLM are detached from the autoregressive path and turned into the fast System 1, while the earlier blocks plus the full LLM remain the slow System 2. System 2's intermediate latent features, taken after block 30, periodically condition System 1, and System 1 additionally reads high-frequency modalities, including a lightweight 3D tokenizer whose point-cloud tokens pass through the shared vision encoder. Training uses $\mathcal{L}_{\mathrm{FiS\text{-}VLA}} = \mathcal{L}_{\mathrm{fast}} + \mathcal{L}_{\mathrm{slow}}$, where $\mathcal{L}_{\mathrm{fast}}$ is a diffusion noise-prediction objective over action chunks and $\mathcal{L}_{\mathrm{slow}}$ is a cross-entropy next-token objective over discrete actions or language, so that the two objectives share one parameter set and each system's representation is preserved.

What would settle it

Run the co-trained model on a standard vision-language reasoning benchmark that the untouched base VLM passes; a large drop relative to the base would contradict the claim that System 2's reasoning is preserved. Alternatively, replace the block-30 latent features given to System 1 with random or frozen features in the same training setup: if success stays near 69 percent, the latent-conditioning mechanism is not the source of the reported gain.

Watch

Extended reading notes

Core claim

The central claim is that a unified vision-language-action model can house both Kahneman-style systems in one network: the slow, deliberate reasoning path keeps the whole transformer, while the final two transformer blocks are repurposed as the fast System 1 execution module that generates continuous action chunks through diffusion. System 2 runs at low frequency on language and 2D images, emitting latent features from block 30 that condition System 1, which runs at high frequency on 2D images, 3D point clouds, and robot state. A dual-aware co-training objective, the sum of a diffusion denoising loss and an autoregressive next-token prediction loss, teaches System 1 to act without destroying System 2's reasoning. The paper reports 69 percent mean success on a 10-task simulated manipulation benchmark versus 61 percent for the previous best method, 68 and 74 percent on two real-world dual-arm platforms versus 59 and 61 percent, and control frequencies of 21.9 Hz at action chunk one and 117.7 Hz at chunk eight.

Load-bearing premise

The intermediate latent features from block 30 of the VLM carry enough task-relevant semantic and spatial information for the final two blocks to produce precise high-frequency actions, and co-training with the diffusion objective does not silently destroy System 2's reasoning ability.

Editorial extensions

If this is right

  • A single network, not two separately trained models, can deliver both deliberate reasoning and real-time control, with a 1:4 System 2 to System 1 frequency ratio sufficient for stable manipulation.
  • Control-frequency gains scale with action chunking: predicting eight actions in one forward pass reaches 117.7 Hz, meaning the policy network stops being the bottleneck for real-time closed-loop robot control.
  • Co-training the autoregressive objective is part of action accuracy, not just a side effect: dropping $\mathcal{L}_{\mathrm{slow}}$ lowers simulated success from 69 to 62 percent.
  • Because the fast executor shares the VLM's pretrained layers, scaling up the underlying VLM should improve execution precision without retraining a separate action head from scratch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not directly evaluate System 2's language or reasoning quality after co-training; a standard VLM benchmark run on the final checkpoint would test the preservation claim outside the action domain.
  • The fixed two-block boundary and fixed 1:4 frequency suggest a natural extension to adaptive gating, where the model decides how many blocks and what update rate a task needs; the authors themselves flag dynamic adaptation as future work.
  • The same 'repurposed blocks inside a pretrained model' trick may transfer to other continuous high-frequency inputs, such as tactile or audio streams, though only 2D images, point clouds, and robot state are tested here.
  • The claimed gain rests on intermediate latent features being genuinely informative; a direct comparison between real block-30 features and frozen or randomized conditioning features would isolate whether the latent guidance or the shared parameters carry the improvement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes FiS-VLA, a 7B vision-language-action model that embeds a fast diffusion-based action module (System 1) inside the final transformer blocks of a pretrained VLM-based System 2, rather than using a separate policy head. System 2 runs at low frequency on language and 2D images, producing intermediate latent features; System 1 runs at high frequency on 2D images, 3D point clouds, and robot state, producing end-effector or joint actions. The two systems are co-trained with a diffusion denoising loss for System 1 and an autoregressive next-token loss for System 2. The model is pretrained on 860K trajectories and fine-tuned on RLBench and two self-collected dual-arm real-world datasets. Reported results include 69% average success on 10 RLBench tasks versus 61% for CogACT, real-world success rates of 68% and 74% on Agilex and AlphaBot versus 59% and 61% for pi0, and control frequencies of 21.9 Hz at action chunk 1 and a claimed 117.7 Hz at action chunk 8.

Significance. If the central architectural hypothesis is substantiated, the paper makes a useful conceptual and engineering contribution: it challenges the common design of attaching a separate fast policy head to a VLM, and it demonstrates a concrete way to share parameters between reasoning and execution within a single foundation model. The paper's strengths include a large-scale pretraining recipe, systematic ablations over shared block count, input modalities, and frequency ratio, evaluation across two real dual-arm platforms with different control modes, and a failure-case appendix that adds practical value. However, the paper's headline conceptual claim is currently confounded by the absence of a matched separate-head control, and the speed claim is presented inconsistently as both measured and theoretical.

major comments (4)
  1. [§3.2, §4.2, Table 1] The paper's central claim is that embedding System 1 inside the VLM's final blocks is what improves manipulation performance. This claim is not tested against a matched separate-head baseline. The comparisons in Table 1 (ManipLLM, OpenVLA, pi0, CogACT) differ simultaneously in base model, pretraining corpus, observation modalities, action parameterization, and training recipe, while the ablations in Figure 3 vary the number of shared blocks but never remove parameter sharing while holding all other factors fixed. The observed +8% simulation gain over CogACT could therefore be explained by the added point-cloud/state inputs, the 860K-trajectory pretraining, or the increased effective capacity, rather than by the Fast-in-Slow architecture. Please add a control policy head of matched capacity and inputs that is external to the VLM but conditioned on the same block-30 intermediate features, 2D image, point cloud, robot state, and noised action, trained with the same pretraining and fine-tuning protocol. This control is load-bearing for the paper's conceptual contribution.
  2. [Abstract, §1, Appendix B.1] The 117.7 Hz figure is presented as an achieved control frequency in the abstract and introduction, but Appendix B.1 states that 'the theoretical control frequency reaches up to 117.7 Hz' when predicting eight future actions. Table 1 reports 21.9 Hz at action chunk one, and no measured end-to-end frequency at chunk eight is reported anywhere. Please clarify whether 117.7 Hz includes the full perception stack (image encoding, point-cloud tokenization, and denoising) or whether it is an idealized arithmetic upper bound derived from action chunking. If it is theoretical, state this clearly in all claims and report a measured closed-loop frequency under the actual deployment conditions.
  3. [§4.3, Table 2, §4.4, Table 3] The real-world evaluation is weaker than the simulation evaluation in several ways that affect the headline '11% real-world improvement' claim. Success is determined by human evaluation, results are reported as single values with no variance or number of repeated rollouts, and the only baseline is pi0; CogACT, which is the stronger simulation baseline, is not evaluated in the real world. Given the small number of tasks and the absence of variance, the cross-method differences in Tables 2 and 3 should be interpreted cautiously. Please report the number of rollouts per task, per-task variance or confidence intervals, and, where feasible, include at least one additional strong baseline or temper the real-world superiority claim accordingly.
  4. [§3.4, §4.2] The paper claims that the dual-aware co-training strategy preserves System 2's reasoning capability while adding action generation, but this is only supported indirectly by the downstream manipulation ablation in which removing Lslow drops success from 69% to 62%. There is no direct evaluation of System 2's language understanding, instruction following, or multimodal reasoning quality before and after co-training. Since the paper's motivation depends on keeping the VLM's pretrained reasoning intact, please add a direct measure of System 2 capability (e.g., performance on a VLM benchmark or instruction-following accuracy) or explicitly limit the claim to the observed manipulation outcomes.
minor comments (7)
  1. [§1] The first sentence of the introduction contains a typo: 'undamental' should be 'fundamental'.
  2. [Abstract and Appendix B.1] The wording of the 117.7 Hz claim should be unified across the abstract, introduction, and appendix so that the measured and theoretical nature of the number is unambiguous.
  3. [Eq. (2)] The symbol D_t is used in Eq. (2) before it is defined; define it before or immediately after the equation.
  4. [Table 3] The caption says 'Left images show the three generalization test scenarios,' but the table as typeset contains no images; either include the images or revise the caption.
  5. [Throughout] There are minor grammar and consistency issues, for example 'Table 2 showcase' should be 'Table 2 showcases', and the baseline name is written both as 'π0' and 'pi0'.
  6. [Abstract and §4] The phrases '8%' and '11%' in the abstract refer to percentage-point differences in success rates; consider using 'percentage points' to avoid ambiguity.
  7. [§3.2] The notation 'θf ⊆ θ' is informal; clarify how the shared-parameter subset relationship is defined.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's success rates and speeds are measured outcomes, not quantities forced by fitted parameters or by self-citations.

full rationale

FiS-VLA is an empirical systems paper. Its central numbers—69% average success on RLBench (Table 1), 68%/74% on the two real-world platforms (Table 2), and 21.9/117.7 Hz control frequency—are obtained from rollouts and forward-pass timing, not derived from the model's own definitions or from a fitted parameter that is then reported as a prediction. The architectural choices (two shared blocks, 1:4 slow/fast frequency ratio, point-cloud plus state plus image inputs for System 1) are selected by ablations (Figure 3; Tables 6–8); this is in-sample model selection, which can inflate reported performance, but it is not circularity because the reported success rate is not the fitting target of those choices by construction. The paper's central causal claim (embedding System 1 inside System 2 is what improves manipulation) is not isolated from a matched separate-head baseline, and gains could partly come from the 860K-trajectory pretraining, the added point-cloud/state inputs, or the co-training objective; however, that is a confound/missing-control issue rather than a reduction of the claim to its inputs. The manuscript does contain self-citations (e.g., the point-cloud encoder 'following [30]' Lift3D, and the diffusion+autoregressive co-training resembles the authors' HybridVLA [32]), but these are not load-bearing: the paper's own No-PC ablation (69 to 61) and Lslow-removal ablation (69 to 62) re-establish the contributions empirically, and no uniqueness theorem or 'forbidden alternative' is imported from the authors' prior work. No equation reduces to itself, and no fitted parameter is renamed as a prediction. Section 5's stated limitation (statically configured shared parameters and frequency ratio) concerns future dynamic adaptation, not circularity. Hence no circular steps, score 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim does not rest on a formal derivation with fitted constants; it rests on standard ML objectives, empirical hyperparameters (block count, frequency ratio, action chunk) chosen by ablation, and domain assumptions about transferability of VLM features and stability of co-training.

free parameters (3)
  • System 1 block count = 2 transformer blocks
    Chosen by ablation on 10 RLBench tasks (Table 6); 2 blocks gives 69%, 1 gives 49%, 4 gives 66%, 8 gives 64%. This hyperparameter directly affects the central architecture and performance claim.
  • System 2:Systemer1 frequency ratio = 1:4
    Chosen by ablation on RLBench (Table 8); 1:4 gives 69% vs 60%, 63%, 61% for 1:1, 1:2, 1:8. The central speed and performance claims depend on this choice.
  • Action chunk size for speed claim = 8
    The 117.7 Hz control frequency is reported with action chunk set to 8 (abstract), chosen to maximize inference speed; Table 9 shows success is stable across chunk sizes 1 to 8.
assumptions (4)
  • standard math Diffusion forward process closed-form noising (Ho et al.)
    Used in Eq. (1); standard DDPM result.
  • domain assumption Intermediate LLM representations are sufficient for low-level action generation
    The paper relies on this to condition System 1 on block-30 latents, citing prior work [24,69]; no direct verification in this paper.
  • domain assumption Pretrained VLM knowledge transfers to robotic manipulation through fine-tuning
    Inherited from the OpenVLA/CogACT paradigm; the paper initializes from Prismatic VLM and fine-tunes on 860K trajectories.
  • domain assumption Summing diffusion and autoregressive losses does not cause catastrophic forgetting
    The dual-aware co-training objective (Eq. 3) assumes the two losses can be optimized jointly with shared blocks; only a single ablation (removing Lslow drops 69% to 62%) supports this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoning." pith.science (2026). https://pith.science/paper/N474HSTI

@misc{pith2026250601953,
  author       = {Pith},
  title        = {Pith review of: Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N474HSTI}},
  note         = {Machine review of arXiv:2506.01953}
}
read the original abstract

Generalized policy and execution efficiency constitute the two critical challenges in robotic manipulation. While recent foundation policies benefit from the common-sense reasoning capabilities of internet-scale pretrained vision-language models (VLMs), they often suffer from low execution frequency. To mitigate this dilemma, dual-system approaches, inspired by Kahneman's theory, have been proposed to leverage a VLM-based System 2 model handling high-level reasoning and a separate System 1 action model ensuring real-time control. However, existing designs maintain both systems as separate models, limiting System 1 from fully leveraging the rich pretrained knowledge from the VLM-based System 2. In this work, we propose Fast-in-Slow (FiS), a unified dual-system vision-language-action (VLA) model that embeds the System 1 execution module within the VLM-based System 2 by partially sharing parameters. This innovative paradigm not only enables high-frequency execution in System 1 but also facilitates coordination between the reasoning and execution components within a single foundation model of System 2. Given their fundamentally distinct roles within FiS-VLA, we design the two systems to incorporate heterogeneous modality inputs alongside asynchronous operating frequencies, enabling both fast and precise manipulation. To enable coordination between the two systems, a dual-aware co-training strategy is proposed that equips System 1 with action generation capabilities while preserving System 2's contextual reasoning representation. For evaluation, FiS-VLA outperforms previous state-of-the-art methods by 8% in simulation and 11% in real-world tasks in terms of average success rate, while achieving a 117.7 Hz control frequency with action chunk set to eight. Project web page: fast-in-slow.github.io.

Figures

Figures reproduced from arXiv: 2506.01953 by the authors.

Figure 1
Figure 1. Overview of FiS-VLA. (a) Unlike previous dual-system VLA methods [1, 2] that attach a separate policy head as System 1, FiS-VLA (b) repurposes the final transformer blocks of an intact VLM as System 1, while retaining the full model for System 2 reasoning. Under this paradigm, FiS-VLA achieves superior performance and high-frequency control, as shown in (c) and (d). and autoregressive action generation, lead to low … view at source ↗
Figure 2
Figure 2. Framework of FiS-VLA. FiS-VLA leverages an intact VLM for System 2 reasoning while repurposing the final transformer blocks of the LLM for System 1 execution module. System 2 handles low-frequency inputs such as 2D images and language instructions and produces intermediate latent features that serve as conditioning information for System 1. Instead of being conditioned solely on these periodically updated high-level… view at source ↗
Figure 3
Figure 3. Ablation study. We investigate the impact of (1) the parameters of System 1’s shared blocks within System 2, (2) different modality inputs to System 1, and (3) the operating frequency ratio between the two systems on final manipulation success rates. 4 Experiments In Section 4.1, we compare the manipulation performance and inference speed of FiS-VLA with prior methods in simulated environments. The effectiveness of … view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Real-world assets and camera configurations. We present visualizations of the real-world assets and camera setups used in the Agilex and AlphaBot dual-arm robot tasks, respectively [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: Ablation studies on action chunk size and input variants of FiS-VLA. (Left) Impact of different action chunk sizes on success rate and inference speed. While increasing action chunk size leads to improved inference speed, success rate remains relatively stable. (Right)…
Figure 6
Figure 6. Figure 6: RLBench visualization. We visualize key frames of the agent’s execution process from the front perspective. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: RLBench visualization. We visualize key frames of the agent’s execution process from the front perspective. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 8
Figure 8. Figure 8: Agilex robot task execution visualization. We visualize key frames of the agent’s execution process from a static exterior view. Task Process ( Agilex Robot ) Pick and place inside Lift ball and place Place bottles at rack Wipe blackboard [PITH_FULL_IMAGE:figures/full…
Figure 9
Figure 9. Figure 9: AlphaBot task execution visualization. We visualize key frames of the agent’s execution process from a static exterior view. D Failure Case Analysis. Through real-world experiments on the AlphaBot platform, we observe four specific failure cases encountered by our prop…
Figure 10
Figure 10. Figure 10: Failure case visualization. We visualize the failure cases observed in four real-world experiments, with key error frames during execution highlighted using red bounding boxes. E Broader Impact Our work proposes a foundation model for robotic manipulation that integra…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots

    cs.RO 2026-07 unverdicted novelty 7.0 of 10

    Embodied.cpp introduces a portable C++ inference runtime with modular layers for deploying VLA and WAM models on heterogeneous robots, reporting 100% and 91% task success on two models plus memory reduction on a WAM b...

  2. Fast and Accurate: An Adaptive VLA Inference Framework through Environment-aware Model Selection

    cs.RO 2026-08 conditional novelty 6.0 of 10

    EMS, a dual-system VLA framework with RL-trained switching, achieves near-large-model success on LIBERO at high effective command rate.

  3. Token-Wise Latent Streaming from Slow Reasoners to Fast Planners for Dynamic Vision Language Navigation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Streaming intermediate hidden states from a slow VLM to a fast flow-matching planner, token by token, improves dynamic social VLN success and reduces observation staleness.

  4. ABot-N1: Toward a General Visual Language Navigation Foundation Model

    cs.CV 2026-07 unverdicted novelty 6.0 of 10

    A slow–fast VLN model that routes five navigation tasks through CoT plus image-space pixel goals reaches SOTA on established and new urban benchmarks, including 77.3% POI arrival.

  5. From Fixed to Free Cameras: Calibration-Free View-Robust Vision-Language-Action Model

    cs.CV 2026-07 conditional novelty 6.0 of 10

    CamVLA decouples robot actions into camera-frame movements and a learned hand-eye pose, enabling calibration-free manipulation under unseen camera viewpoints from a single RGB image.

  6. AsyncMDE: Real-Time Monocular Depth Estimation via Asynchronous Spatial Memory

    cs.RO 2026-03 conditional novelty 6.0 of 10

    A lightweight fast path fusing foundation-model spatial memory reaches 237 FPS monocular depth, recovering 77% of the accuracy gap with bounded lag degradation and 161 FPS on Jetson Orin.

  7. TIDAL: Temporally Interleaved Diffusion and Action Loop for High-Frequency VLA Control

    cs.RO 2026-01 conditional novelty 6.0 of 10

    TIDAL raises VLA control feedback from ~2.4 Hz to ~9 Hz by caching semantic intent and interleaving one-step flow generation with execution, doubling dynamic interception success in simulation.

  8. RwoR: Generating Robot Demonstrations from Human Hand Collection for Policy Learning without Robot

    cs.RO 2025-07 conditional novelty 6.0 of 10

    A generative model and wrist camera turn human hand videos into robot gripper demonstrations that train manipulation policies at success rates close to those trained on real gripper data.

  9. Emergent Compositional Skills in Mixture-of-Experts VLAs

    cs.RO 2026-07 conditional novelty 5.0 of 10

    A VLA with a whole-forward-pass LoRA-MoE action head shows qualitatively reusable, skill-like expert behavior on LIBERO-10, but the claimed performance parity with a monolithic baseline is unquantified.

  10. Lift3D-VLA: Lifting VLA Models to 3D Geometry and Dynamics-Aware Manipulation

    cs.RO 2026-07 conditional novelty 5.0 of 10

    Lift3D-VLA integrates 3D point cloud encoding and temporal action modeling into Vision-Language-Action models, achieving higher success rates on simulated and real-world robotic manipulation tasks.

  11. Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey

    cs.RO 2025-10 conditional novelty 4.0 of 10

    A survey that groups VLA efficiency techniques into four categories: model architecture, perception features, action generation, and training/inference strategies.

Reference graph

Works this paper leans on

120 extracted references · 17 canonical work pages · cited by 11 Pith papers

  1. [1]

    Hirt: Enhancing robotic control with hierarchical robot transformers

    Jianke Zhang, Yanjiang Guo, Xiaoyu Chen, Yen-Jen Wang, Yucheng Hu, Chengming Shi, and Jianyu Chen. Hirt: Enhancing robotic control with hierarchical robot transformers. arXiv preprint arXiv:2410.05273, 2024

    arXiv 2024

  2. [2]

    Towards synergistic, generalized, and efficient dual-system for robotic manipulation, 2025

    Qingwen Bu, Hongyang Li, Li Chen, Jisong Cai, Jia Zeng, Heming Cui, Maoqing Yao, and Yu Qiao. Towards synergistic, generalized, and efficient dual-system for robotic manipulation, 2025

    2025

  3. [3]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023

    2023

  4. [4]

    Diffusion transformer policy

    Zhi Hou, Tianyi Zhang, Yuwen Xiong, Hengjun Pu, Chengyang Zhao, Ronglei Tong, Yu Qiao, Jifeng Dai, and Yuntao Chen. Diffusion transformer policy. arXiv preprint arXiv:2410.15959, 2024

    arXiv 2024

  5. [5]

    Rvt: Robotic view transformer for 3d object manipulation

    Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox. Rvt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning, pages 694–710. PMLR, 2023

    2023

  6. [6]

    Cliport: What and where pathways for robotic manipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on robot learning, pages 894–906. PMLR, 2022

    2022

  7. [7]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024

    arXiv 2024

  8. [9]

    Cot-vla: Visual chain-of-thought reasoning for vision-language-action models

    Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, et al. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. arXiv preprint arXiv:2503.22020, 2025

    arXiv 2025

Show all 120 references
  1. [10]

    Rt-h: Action hierarchies using language

    Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. Rt-h: Action hierarchies using language. arXiv preprint arXiv:2403.01823, 2024. 10

    2024 arXiv

  2. [11]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023

    2023 arXiv

  3. [12]

    Do as i can, not as i say: Grounding language in robotic affordances

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022

    2022 arXiv

  4. [13]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716...

    2022

  5. [14]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  6. [15]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  7. [16]

    Prismatic vlms: Investigating the design space of visually-conditioned language models

    Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. arXiv preprint arXiv:2402.07865, 2024

    2024 arXiv

  8. [17]

    Lidar-llm: Exploring the potential of large language models for 3d lidar understanding

    Senqiao Yang, Jiaming Liu, Renrui Zhang, Mingjie Pan, Ziyu Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Hongsheng Li, Yandong Guo, et al. Lidar-llm: Exploring the potential of large language models for 3d lidar understanding. In Proceedings of the AAAI Conference on Artificial Intell...

    2025

  9. [18]

    Mavis: Mathematical visual instruction tuning with an automatic data engine

    Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Ziyu Guo, Shicheng Li, Yichi Zhang, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, et al. Mavis: Mathematical visual instruction tuning with an automatic data engine. arXiv preprint arXiv:2407.08739, 2024

    2024 arXiv

  10. [20]

    Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Mill...

    2024

  11. [21]

    Thinking, fast and slow

    Daniel Kahneman. Thinking, fast and slow. Farrar, Straus and Giroux, 2011

    2011

  12. [22]

    Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation

    Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al. Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation. arXiv preprint arXiv:2411.19650, 2024

    2024 arXiv

  13. [23]

    pi0: A vision-language-action flow model for general robot control

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. pi0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024

    arXiv

  14. [24]

    Gr00t n1: An open foundation model for generalist humanoid robots

    Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025

    2025 arXiv

  15. [25]

    Helix: A vision-language-action model for generalist humanoid control

    figureai. Helix: A vision-language-action model for generalist humanoid control. https: //www.figure.ai/news/helix. Accessed 2025.5.7

    2025

  16. [26]

    An fmri investigation of emotional engagement in moral judgment

    Joshua D Greene, R Brian Sommerville, Leigh E Nystrom, John M Darley, and Jonathan D Cohen. An fmri investigation of emotional engagement in moral judgment. Science, 293(5537):2105–2108, 2001

    2001

  17. [27]

    The neural bases of cognitive conflict and control in moral judgment

    Joshua D Greene, Leigh E Nystrom, Andrew D Engell, John M Darley, and Jonathan D Cohen. The neural bases of cognitive conflict and control in moral judgment. Neuron, 44(2):389–400, 2004

    2004

  18. [28]

    3d diffusion policy

    Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy. arXiv preprint arXiv:2403.03954, 2024

    2024 arXiv

  19. [29]

    Spatialvla: Exploring spatial representations for visual-language-action model

    Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, JiaYuan Gu, Bin Zhao, Dong Wang, et al. Spatialvla: Exploring spatial representations for visual-language-action model. arXiv preprint arXiv:2501.15830, 2025

    2025 arXiv

  20. [30]

    Lift3d foundation policy: Lifting 2d large-scale pretrained models for robust 3d robotic manipulation

    Yueru Jia, Jiaming Liu, Sixiang Chen, Chenyang Gu, Zhilue Wang, Longzan Luo, Lily Lee, Pengwei Wang, Zhongyuan Wang, Renrui Zhang, et al. Lift3d foundation policy: Lifting 2d large-scale pretrained models for robust 3d robotic manipulation. arXiv preprint arXiv:2411.18623, 2024

    2024 arXiv

  21. [31]

    Rdt-1b: a diffusion foundation model for bimanual manipulation

    Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864, 2024. 12

    2024 arXiv

  22. [32]

    Hybridvla: Collaborative diffusion and autoregression in a unified vision-language-action model

    Jiaming Liu, Hao Chen, Pengju An, Zhuoyang Liu, Renrui Zhang, Chenyang Gu, Xiaoqi Li, Ziyu Guo, Sixiang Chen, Mengzhen Liu, et al. Hybridvla: Collaborative diffusion and autoregression in a unified vision-language-action model. arXiv preprint arXiv:2503.10631, 2025

    2025 arXiv

  23. [33]

    Open x-embodiment: Robotic learning datasets and rt-x models

    Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Ab- hishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023

    arXiv

  24. [34]

    Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation

    Kun Wu, Chengkai Hou, Jiaming Liu, Zhengping Che, Xiaozhu Ju, Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang, et al. Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation. arXiv preprint arXiv:2412.13877, 2024

    2024 arXiv

  25. [35]

    Rlbench: The robot learning benchmark & learning environment

    Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019–3026, 2020

    2020

  26. [36]

    Learning dexterous in-hand manipulation

    OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3–20, 2020

    2020

  27. [37]

    End-to- end affordance learning for robotic manipulation

    Yiran Geng, Boshi An, Haoran Geng, Yuanpei Chen, Yaodong Yang, and Hao Dong. End-to- end affordance learning for robotic manipulation. In International Conference on Robotics and Automation (ICRA), 2023

    2023

  28. [38]

    Robotic grasping using deep reinforcement learning

    Shirin Joshi, Sulabh Kumra, and Ferat Sahin. Robotic grasping using deep reinforcement learning. In 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), pages 1461–1466. IEEE, 2020

    2020

  29. [39]

    Mastering visual continuous control: Improved data-augmented reinforcement learning

    Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021

    2021 arXiv

  30. [40]

    Behavioral cloning from observation

    Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954, 2018

    2018 arXiv

  31. [41]

    Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation

    Zipeng Fu, Tony Z Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02117 , 2024

    2024 arXiv

  32. [42]

    Rt-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022

    2022 arXiv

  33. [43]

    Crayonrobo: Toward generic robot manipulation via crayon visual prompting

    Xiaoqi Li, Lingyun Xu, Jiaming Liu, Mingxu Zhang, Jiahui Xu, Siyuan Huang, Iaroslav Ponomarenko, Yan Shen, Shanghang Zhang, and Hao Dong. Crayonrobo: Toward generic robot manipulation via crayon visual prompting. 2024

    2024

  34. [44]

    Autonomous interactive correction mllm for robust robotic manipulation

    Chuyan Xiong, Chengyu Shen, Xiaoqi Li, Kaichen Zhou, Jiaming Liu, Ruiping Wang, and Hao Dong. Autonomous interactive correction mllm for robust robotic manipulation. In 8th Annual Conference on Robot Learning, 2024

    2024

  35. [45]

    Naturalvlm: Leveraging fine-grained natural language for affordance-guided visual manipulation

    Ran Xu, Yan Shen, Xiaoqi Li, Ruihai Wu, and Hao Dong. Naturalvlm: Leveraging fine-grained natural language for affordance-guided visual manipulation. arXiv preprint arXiv:2403.08355, 2024

    2024 arXiv

  36. [46]

    Self-corrected multimodal large language model for end-to-end robot manipulation

    Jiaming Liu, Chenxuan Li, Guanqun Wang, Lily Lee, Kaichen Zhou, Sixiang Chen, Chuyan Xiong, Jiaxin Ge, Renrui Zhang, and Shanghang Zhang. Self-corrected multimodal large language model for end-to-end robot manipulation. arXiv preprint arXiv:2405.17418, 2024. 13

    2024 arXiv

  37. [47]

    V oxposer: Composable 3d value maps for robotic manipulation with language models

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023

    2023 arXiv

  38. [48]

    Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation

    Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652, 2024

    2024 arXiv

  39. [49]

    Rt-2: Vision-language- action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language- action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023

    2023 arXiv

  40. [50]

    Manipllm: Embodied multimodal large language model for object-centric robotic manipulation

    Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2024

  41. [51]

    Fast: Efficient action tokenization for vision-language-action models

    Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.09747, 2025

    2025 arXiv

  42. [52]

    Robomamba: Efficient vision-language-action model for robotic reasoning and manipulation

    Jiaming Liu, Mengzhen Liu, Zhenyu Wang, Pengju An, Xiaoqi Li, Kaichen Zhou, Sen- qiao Yang, Renrui Zhang, Yandong Guo, and Shanghang Zhang. Robomamba: Efficient vision-language-action model for robotic reasoning and manipulation. Advances in Neural Information Processing Syste...

    2024

  43. [53]

    Manipvqa: Injecting robotic affordance and phys- ically grounded information into multi-modal large language models

    Siyuan Huang, Iaroslav Ponomarenko, Zhengkai Jiang, Xiaoqi Li, Xiaobin Hu, Peng Gao, Hongsheng Li, and Hao Dong. Manipvqa: Injecting robotic affordance and phys- ically grounded information into multi-modal large language models. arXiv preprint arXiv:2403.11289, 2024

    2024 arXiv

  44. [54]

    Vision-language foundation models as effective robot imitators

    Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators. arXiv preprint arXiv:2311.01378, 2023

    2023 arXiv

  45. [55]

    Unleashing large-scale video generative pre-training for visual robot manipulation

    Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. arXiv preprint arXiv:2312.13139, 2023

    2023 arXiv

  46. [56]

    Pointvla: Injecting the 3d world into vision-language-action models, 2025

    Chengmeng Li, Junjie Wen, Yan Peng, Yaxin Peng, Feifei Feng, and Yichen Zhu. Pointvla: Injecting the 3d world into vision-language-action models, 2025

    2025

  47. [57]

    Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot

    Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Chenxi Wang, Junbo Wang, Haoyi Zhu, and Cewu Lu. Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot. Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition@ CoRL2023, 3:5, 2023

    2023

  48. [58]

    Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems, 2025

    AgiBot-World-Contributors, Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, Shu Jiang, Yuxin Jiang, Cheng Jing, Hongyang Li, Jialu Li, Chiming Liu, Yi Liu, Yuxiang Lu, Jianlan Luo, Ping Luo, Yao Mu, Yuehan Niu, Yix...

    2025

  49. [59]

    From llms to actions: Latent codes as bridges in hierarchical robot control, 2024

    Yide Shentu, Philipp Wu, Aravind Rajeswaran, and Pieter Abbeel. From llms to actions: Latent codes as bridges in hierarchical robot control, 2024

    2024

  50. [60]

    A dual process vla: Efficient robotic manipulation leveraging vlm, 2024

    ByungOk Han, Jaehong Kim, and Jinhyeok Jang. A dual process vla: Efficient robotic manipulation leveraging vlm, 2024. 14

    2024

  51. [61]

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y . Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin L...

    2025

  52. [62]

    Dexvla: Vision-language model with plug-in diffusion expert for general robot control

    Junjie Wen, Yichen Zhu, Jinming Li, Zhibin Tang, Chaomin Shen, and Feifei Feng. Dexvla: Vision-language model with plug-in diffusion expert for general robot control. arXiv preprint arXiv:2502.05855, 2025

    2025 arXiv

  53. [63]

    Knowledge insulat- ing vision-language-action models: Train fast, run fast, generalize better

    Danny Driess, Jost Tobias Springenberg, Brian Ichter, Lili Yu, Adrian Li-Bell, Karl Pertsch, Allen Z Ren, Homer Walke, Quan Vuong, Lucy Xiaoyang Shi, et al. Knowledge insulat- ing vision-language-action models: Train fast, run fast, generalize better. arXiv preprint arXiv:2505...

    2025 arXiv

  54. [64]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In International Conference on Computer Vision (ICCV), 2023

    2023

  55. [65]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  56. [66]

    No time to train: Empowering non-parametric networks for few-shot 3d scene segmentation

    Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Jiaming Liu, Han Xiao, Chaoyou Fu, Hao Dong, and Peng Gao. No time to train: Empowering non-parametric networks for few-shot 3d scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  57. [67]

    Pointnet: Deep learning on point sets for 3d classification and segmentation

    Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017

    2017

  58. [68]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  59. [69]

    Deer-vla: Dynamic inference of multimodal large language models for efficient robot execution

    Yang Yue, Yulin Wang, Bingyi Kang, Yizeng Han, Shenzhi Wang, Shiji Song, Jiashi Feng, and Gao Huang. Deer-vla: Dynamic inference of multimodal large language models for efficient robot execution. arXiv preprint arXiv:2411.02359, 2024

    2024 arXiv

  60. [70]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  61. [71]

    The open motion planning library

    Ioan A Sucan, Mark Moll, and Lydia E Kavraki. The open motion planning library. IEEE Robotics & Automation Magazine, 19(4):72–82, 2012

    2012

  62. [72]

    Perceiver-actor: A multi-task transformer for robotic manipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022

    2022

  63. [73]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  64. [74]

    QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation

    Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv:1806.1...

    2018 arXiv

  65. [75]

    Bridge data: Boosting generalization of robotic skills with cross-domain datasets

    Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.13396, 2021. 15

    2021 arXiv

  66. [76]

    Bridgedata v2: A dataset for robot learning at scale, 2023

    Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale, 2023

    2023

  67. [77]

    Latent plans for task agnostic offline reinforcement learning

    Erick Rosete-Beas, Oier Mees, Gabriel Kalweit, Joschka Boedecker, and Wolfram Burgard. Latent plans for task agnostic offline reinforcement learning. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022

    2022

  68. [78]

    Grounding language with visual affordances over unstructured data

    Oier Mees, Jessica Borja-Diaz, and Wolfram Burgard. Grounding language with visual affordances over unstructured data. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK, 2023

    2023

  69. [79]

    Shivin Dass, Jullian Yapeter, Jesse Zhang, Jiahui Zhang, Karl Pertsch, Stefanos Nikolaidis, and Joseph J. Lim. CLVR jaco play dataset, 2023

    2023

  70. [80]

    Multi-stage cable routing through hierarchical imitation learning

    Jianlan Luo, Charles Xu, Xinyang Geng, Gilbert Feng, Kuan Fang, Liam Tan, Stefan Schaal, and Sergey Levine. Multi-stage cable routing through hierarchical imitation learning. arXiv preprint arXiv:2307.08927, 2023

    2023 arXiv

  71. [81]

    RoboTurk: A crowdsourcing platform for robotic skill learning through imitation

    Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, Silvio Savarese, and Li Fei-Fei. RoboTurk: A crowdsourcing platform for robotic skill learning through imitation. CoRR, abs/1811.02790, 2018

    2018 arXiv

  72. [82]

    Viola: Imitation learning for vision-based manipulation with object proposal priors, 2023

    Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors, 2023

    2023

  73. [83]

    Berkeley UR5 demonstration dataset

    Lawrence Yunliang Chen, Simeon Adebola, and Ken Goldberg. Berkeley UR5 demonstration dataset. https://sites.google.com/view/berkeley-ur5/home

  74. [84]

    Train offline, test online: A real robot learning benchmark, 2023

    Gaoyue Zhou, Victoria Dean, Mohan Kumar Srirama, Aravind Rajeswaran, Jyothish Pari, Kyle Hatch, Aryan Jain, Tianhe Yu, Pieter Abbeel, Lerrel Pinto, Chelsea Finn, and Abhinav Gupta. Train offline, test online: A real robot learning benchmark, 2023

    2023

  75. [85]

    Interactive language: Talking to robots in real time

    Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023

    2023

  76. [86]

    Hydra: Hybrid robot actions for imitation learning

    Suneel Belkhale, Yuchen Cui, and Dorsa Sadigh. Hydra: Hybrid robot actions for imitation learning. arxiv, 2023

    2023

  77. [87]

    Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation

    Yifeng Zhu, Peter Stone, and Yuke Zhu. Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation. IEEE Robotics and Automation Letters, 7(2):4126–4133, 2022

    2022

  78. [88]

    From play to policy: Conditional behavior generation from uncurated robot data

    Zichen Jeff Cui, Yibin Wang, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:2210.10047, 2022

    2022 arXiv

  79. [89]

    Minho Heo, Youngwoon Lee, Doohyun Lee, and Joseph J. Lim. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation. In Robotics: Science and Systems, 2023

    2023

  80. [90]

    ucsd kitchens Dataset

    Ge Yan, Kris Wu, and Xiaolong Wang. ucsd kitchens Dataset. August 2023

    2023

  81. [91]

    Learning and retrieval from prior data for skill-based imitation learning

    Soroush Nasiriany, Tian Gao, Ajay Mandlekar, and Yuke Zhu. Learning and retrieval from prior data for skill-based imitation learning. In Conference on Robot Learning (CoRL), 2022

    2022

  82. [92]

    Robot learning on the job: Human-in-the-loop autonomy and learning during deployment

    Huihan Liu, Soroush Nasiriany, Lance Zhang, Zhiyao Bao, and Yuke Zhu. Robot learning on the job: Human-in-the-loop autonomy and learning during deployment. In Robotics: Science and Systems (RSS), 2023

    2023

  83. [93]

    Shared Control Templates for Assistive Robotics

    Gabriel Quere, Annette Hagengruber, Maged Iskandar, Samuel Bustamante, Daniel Leidner, Freek Stulp, and Joern V ogel. Shared Control Templates for Assistive Robotics. In2020 IEEE International Conference on Robotics and Automation (ICRA), page 7, Paris, France, 2020. 16

    2020

  84. [94]

    Multi-resolution sensing for real-time control with vision-language models

    Saumya Saxena, Mohit Sharma, and Oliver Kroemer. Multi-resolution sensing for real-time control with vision-language models. In 7th Annual Conference on Robot Learning, 2023

    2023

  85. [95]

    MUTEX: Learning unified policies from multimodal task specifications

    Rutav Shah, Roberto Martín-Martín, and Yuke Zhu. MUTEX: Learning unified policies from multimodal task specifications. In 7th Annual Conference on Robot Learning, 2023

    2023

  86. [96]

    Fanuc manipulation: A dataset for learning-based manipulation with fanuc mate 200id robot

    Xinghao Zhu, Ran Tian, Chenfeng Xu, Mingyu Ding, Wei Zhan, and Masayoshi Tomizuka. Fanuc manipulation: A dataset for learning-based manipulation with fanuc mate 200id robot. 2023

    2023

  87. [97]

    Structured world models from human videos

    Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured world models from human videos. CoRL, 2023

    2023

  88. [98]

    Bc-z: Zero-shot task generalization with robotic imitation learning

    Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991–1002. PMLR, 2022

    2022

  89. [99]

    Fmb: a functional manipulation benchmark for generalizable robotic learning

    Jianlan Luo, Charles Xu, Fangchen Liu, Liam Tan, Zipeng Lin, Jeffrey Wu, Pieter Abbeel, and Sergey Levine. Fmb: a functional manipulation benchmark for generalizable robotic learning. arXiv preprint arXiv:2401.08553, 2024

    2024 arXiv

  90. [100]

    On bringing robots home, 2023

    Nur Muhammad Mahi Shafiullah, Anant Rai, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, and Lerrel Pinto. On bringing robots home, 2023

    2023

  91. [101]

    Lee, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg

    Michelle A. Lee, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks, 2019

    2019

  92. [102]

    Robocook: Long-horizon elasto-plastic object manipulation with diverse tools, 2023

    Haochen Shi, Huazhe Xu, Samuel Clarke, Yunzhu Li, and Jiajun Wu. Robocook: Long-horizon elasto-plastic object manipulation with diverse tools, 2023

    2023

  93. [103]

    Maniskill2: A unified benchmark for generalizable manipulation skills, 2023

    Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskill2: A unified benchmark for generalizable manipulation skills, 2023

    2023

  94. [104]

    Real-world robot learning with masked visual pre-training

    Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In Conference on Robot Learning, pages 416–426. PMLR, 2023

    2023

  95. [105]

    Lhmanip: A dataset for long-horizon language-grounded manipulation tasks in cluttered tabletop environments

    Federico Ceola, Lorenzo Natale, Niko Sünderhauf, and Krishan Rana. Lhmanip: A dataset for long-horizon language-grounded manipulation tasks in cluttered tabletop environments. arXiv preprint arXiv:2312.12036, 2023

    2023 arXiv

  96. [106]

    Robohive: A unified framework for robot learning

    Vikash Kumar, Rutav Shah, Gaoyue Zhou, Vincent Moens, Vittorio Caggiano, Abhishek Gupta, and Aravind Rajeswaran. Robohive: A unified framework for robot learning. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023

    2023

  97. [107]

    A reduction of imitation learning and structured prediction to no-regret online learning

    Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. InProceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627–635. JMLR Workshop and Con...

    2011

  98. [108]

    Learning fine-grained bimanual manipulation with low-cost hardware

    Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023. 17 Appendix A Additional Dataset Details. In this section, the construction of real-world datasets for large-s...

    2023 arXiv

  99. [109]

    The robot uses both arms to pick up two objects according to a language command and place them into a container

    Pick objects and place in basket. The robot uses both arms to pick up two objects according to a language command and place them into a container. This task assesses the model’s understanding of spatial positioning. 19 (b) AlphaBotdual-arm robot(a)Agilexdual-arm robotRealSense...

  100. [110]

    The robot must synchronize both arms to grasp a ball held between the grippers and transport it without slippage

    Lift ball and place in basket. The robot must synchronize both arms to grasp a ball held between the grippers and transport it without slippage. This task evaluates dual-arm coordination

  101. [111]

    Each arm grasps a bottle from its side, rotates it, and aligns it parallel to the rack

    Place bottles at rack. Each arm grasps a bottle from its side, rotates it, and aligns it parallel to the rack. This task tests inter-object relationship reasoning and precise rotational manipulation

  102. [112]

    One arm holds the board while the other erases red marker using an eraser

    Wipe blackboard. One arm holds the board while the other erases red marker using an eraser. This setup tests precise, coordinated actions in dual-arm scenarios. AlphaBot robot setup. As shown in Table 5, the AlphaBot leverages two 7-DoF arms mounted on a mobile base. As shown ...

  103. [113]

    The robot uses its left arm to pick up a bowl and its right arm to pick up an object, placing the object into the bowl

    Pick bowl and place object. The robot uses its left arm to pick up a bowl and its right arm to pick up an object, placing the object into the bowl. This task involves coordinated dual-arm manipulation, where each arm performs distinct, asymmetric roles

  104. [114]

    The right arm picks up an object and hands it to the left arm

    Handover object and place. The right arm picks up an object and hands it to the left arm. The arms must avoid collisions and ensure proper grasp alignment. The left arm then places the object into a plate. This task serves as a comprehensive benchmark for evaluating the model’...

  105. [115]

    The robot grasps a cup handle with its right arm, rotates it to pour water into another cup, then moves the receiving cup to a coaster

    Pour water and move cup. The robot grasps a cup handle with its right arm, rotates it to pour water into another cup, then moves the receiving cup to a coaster. This task combines high-precision 20 Figure 5: Ablation studies on action chunk size and input variants of FiS-VLA. ...

  106. [116]

    The robot folds a deformable towel using both arms, then places it into a bucket

    Fold towel and place in bucket. The robot folds a deformable towel using both arms, then places it into a bucket. This task evaluates coordinated manipulation of deformable objects. B Additional Quantitative Results B.1 Action chunking for robust and high-Frequency robot contr...

  107. [117]

    We observe that the performance of FiS-VLA remains stable across different values of H, while the control frequency increases proportionally. Notably, when predicting eight future actions in a single step, the theoretical control frequency reaches up to 117.7 Hz, demonstrating...

  108. [118]

    The results are presented in Figure 5 and Table 10

    We refer to these configurations as the input variants of FiS-VLA. The results are presented in Figure 5 and Table 10. In Variant 1, System 2 receives language instructions, 2D images, and 3D point clouds, while System 1 takes 2D images and robot state as input. This configura...

  109. [119]

    The first case involves a bimanual collision during the Handover object and place into plate task. The left and right arms interfere with each other while attempting to transfer the object, indicating insufficient inter-arm motion coordination and suboptimal wrist camera placement

  110. [120]

    The predicted joint positions fail to control gripper contact with the towel, revealing the difficulty of height prediction when dealing with thin, deformable objects

    The second case, observed in the Fold towel and place in bucket task, is related to an error in manipulation height. The predicted joint positions fail to control gripper contact with the towel, revealing the difficulty of height prediction when dealing with thin, deformable objects

  111. [121]

    The robot mispredicts the location of the banana, resulting in a failed grasp attempt

    The third case, from the Pick bowl and place object task, reflects a failure in manipulation position. The robot mispredicts the location of the banana, resulting in a failed grasp attempt

  112. [122]

    The right arm rotates the object into an unsuitable orientation, preventing the left arm from executing a stable handover grasp

    The fourth case presents a handover rotation error in the Handover object and place into plate task. The right arm rotates the object into an unsuitable orientation, preventing the left arm from executing a stable handover grasp. These issues can be mitigated by collecting mor...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.