Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

RationalVLA: A Rational Vision-Language-Action Model with Dual System

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a dual-system VLA with <ACT> and <REJ> tokens can reject defective instructions and execute unseen ones, outperforming single policies and VLM-as-identifier pipelines on the new RAMA benchmark.

desk verdict RAMA is a genuinely useful benchmark and the <REJ> token is a sensible interface, but the headline comparison on RAMA is confounded because the VLM-as-identifier baselines were never trained on the RAMA distribution; the paper still deserves review with a control experiment. read the letter →

arxiv 2506.10826 v2 pith:CS4PPT3I submitted 2025-06-12 cs.RO

Wenxuan Song , Jiayi Chen , Wenxue Li , Xu He , Han Zhao , Can Cui , Pengxiang Ding Shiyan Su , Feilong Tang
show 7 more authors
Xuelian Cheng Donglin Wang Zongyuan Ge Xinhu Zheng Zhe Liu Hesheng Wang Haoang Li
This is my paper
classification cs.RO
keywords vision-language-actionmodeldefectiveinstructionrejectiondual-systemarchitecturelatentactionembeddingRAMAbenchmarklong-horizonrobotmanipulationlanguage-conditioned<ACT>/<REJ>tokens
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that language-conditioned manipulation fails in real deployments when instructions are ambiguous, infeasible, irrelevant, or unsafe, and that existing benchmarks do not test this. It introduces RAMA, a benchmark with 14,412 instructions that mix unseen-but-executable commands with defective ones a robot should refuse, grouped into six defect dimensions. It then presents RationalVLA, a dual-system model in which a high-level multimodal language model emits one of two learned tokens: , whose hidden embedding is passed as a latent goal to a low-level diffusion policy, or , which halts action output. On RAMA the model improves average completed-task length from 1.30 to 2.26 and last-task success by 14.5 percentage points over the strongest baseline, while staying competitive on standard CALVIN tasks. If the results hold, the paper establishes that rejecting defective instructions can be learned end-to-end inside an action policy rather than bolted on as a separate filter.

What carries the argument

The load-bearing mechanism is a two-token vocabulary extension on a pretrained vision-language model. The <ACT> token is trained to appear in the model's reply when an instruction is executable; the hidden-state embedding of that token is projected into the conditioning space of a pretrained 3D diffusion policy, which then generates end-effector trajectories. The <REJ> token is trained to appear for defective instructions, and its embedding is projected to an all-zero tensor that halts the policy. The two tokens are learned end-to-end with LoRA, a chat-style supervised fine-tuning objective, and a CLIP-style alignment loss that keeps the <ACT> embedding close to the text description of the ground-truth task.

What would settle it

Run the RAMA evaluation with defective instructions collected from ordinary human users describing impossible tasks in their own words, and check whether RationalVLA's rejection accuracy and average-length advantage over the strongest VLM-as-identifier baseline survive when the defective examples no longer carry template or language-model generation cues.

Watch

Extended reading notes

Core claim

The paper claims that a dual-system VLA can handle the full spectrum of instructions—seen, unseen, and defective—only when the high-level language model can communicate action intent to the low-level policy through a continuous latent embedding instead of through text alone. RationalVLA realizes this by expanding the VLM vocabulary with <ACT> and <REJ> tokens: <ACT> passes a projected hidden-state embedding to the pretrained 3D diffusion policy as conditioning, while <REJ> projects to a zero tensor that stops the policy. The paper reports that on its RAMA test split this design reaches 74.3, 58.3, 42.3, 30.0, and 20.7 percent success for one to five consecutive tasks, compared with 64.3, 33.1, 16.0, 10.0, and 6.2 for the strongest VLM-as-identifier baseline built from the same low-level policy, raising average completed-task length from 1.30 to 2.26. It also reports 48.0 percent fifth-task success on the standard CALVIN ABC-to-D split, above the 41.2 percent of the policy-only baseline, and real-world average success of 85.0 percent versus 16.7 percent for the policy baseline.

Load-bearing premise

The central comparison rests on the assumption that RAMA's defective instructions, built by hindsight relabeling of CALVIN trajectories through part-of-speech replacement and automatic language-model generation, resemble the defective instructions a deployed robot actually encounters, and that the 159-instruction held-out test set is large enough to support the reported gap.

Editorial extensions

If this is right

  • Deployed robots can decline infeasible commands instead of attempting them, preserving the scene for later tasks and avoiding cascading failures in long-horizon rollouts.
  • A VLM's language understanding can be transmitted to a specialized policy through a learned latent embedding, so the policy does not need to re-learn language from scratch.
  • RAMA offers a standard testbed in which models must simultaneously generalize to unseen phrasing and reject defective phrasing, making the two abilities measurable rather than assumed.
  • On the classic CALVIN benchmark, the dual-system model also improves fifth-task success to 48.0 percent from 41.2 percent, indicating that rejection training does not sacrifice ordinary manipulation performance.
  • Real-world trials with a physical arm show the same rejection and unseen-instruction behavior transfers outside simulation, with average success of 85.0 percent versus 16.7 percent for the policy-only baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same two-token interface could transfer to navigation, mobile manipulation, or human-robot collaboration, where out-of-scope commands also need to be refused without derailing subsequent goals.
  • Editorial inference: because the paper's evaluation places the defective instruction only at the start of each rollout, a stronger test would interleave defective instructions at random positions to measure how rejection behaves under accumulated state changes.
  • Editorial inference: the <REJ> training objective is essentially a visual-question-answering task, so it may reduce object hallucination in the high-level model on general VQA benchmarks; the paper hints at this but does not measure it.
  • Editorial inference: if the model exposed a confidence score for rejection, users could tune the refusal threshold to trade safety against over-rejection, a practical control the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces RAMA, a benchmark that augments CALVIN with defective instructions across six dimensions, and RationalVLA, a dual-system vision-language-action model that couples a high-level MLLM with a low-level 3D Diffuser Actor policy through learnable <ACT> and <REJ> tokens. The model is trained on a mixture of CALVIN and RAMA data and evaluated on RAMA rollouts, standard CALVIN ABC→D tasks, and real-world manipulation trials. The main empirical claims are that RationalVLA improves average task length from 1.30 to 2.26 and 5/5 success from 6.2% to 20.7% over the GPT-4o+3DDA baseline on RAMA, while remaining competitive on standard CALVIN tasks. The paper also contributes a 14k-sample dataset, a six-dimensional taxonomy of defective instructions, and an evaluation protocol that places a defective instruction before a CALVIN long-horizon rollout.

Significance. If the empirical claims hold, the benchmark and model address a real and understudied problem: robots must reject infeasible or defective natural-language instructions rather than blindly executing them. The dual-system architecture is attractive in its simplicity—two extra tokens serve as the high-level/low-level interface—and the paper gives credit for the dataset, the public project page, and real-world validation. The ablation in Table IV, showing a 1.22 average-length drop without the CLIP loss, provides some evidence that the latent-alignment mechanism matters. However, the headline RAMA comparison is confounded by training-data exposure, the test set is small and lacks error bars, and the protocol contains an internal inconsistency about the effect of defective instructions. These issues prevent the paper, as written, from establishing its central claim cleanly.

major comments (4)
  1. [V-A and Table II] The headline RAMA comparison is confounded by training-data exposure. RationalVLA is trained end-to-end on 14,253 RAMA samples (Section III-C, Table I), whereas the VLM-as-identifier baselines (LLaVA+3DDA and GPT-4o+3DDA) use a prompt-engineered VLM that is not fine-tuned on RAMA, as stated in Section V-A. The reported gains of 2.26 vs. 1.30 in average length and 20.7% vs. 6.2% on 5/5 success therefore do not isolate the dual-system <ACT>/<REJ> architecture; they could reflect simple exposure to the RAMA defective-instruction distribution. Please add controls that train the identifier on RAMA (for example, a LoRA-fine-tuned LLaVA or a fine-tuned VLM used as identifier), or evaluate RationalVLA trained only on CALVIN, so that the architectural benefit is separated from the data benefit.
  2. [III-D and Fig. 8] The evaluation protocol is internally inconsistent about the effect of defective instructions. Section III-D states that defective tasks are "designed not to affect the proper initiation of subsequent tasks" and are excluded from success-rate calculations, but Fig. 8 shows a rollout in which an erroneously executed defective instruction moves the red block and causes the next task to fail. Please clarify whether the "not affect" property is intended to hold only when the model correctly rejects the instruction, or is a guarantee of the benchmark construction, and specify how the protocol treats rollouts in which a defective instruction is executed. As written, the intended semantics of the benchmark are ambiguous.
  3. [V-B, Tables I-II, and Fig. 7] The RAMA evaluation rests on a single 159-instruction held-out test set with no confidence intervals, bootstrap estimates, or multiple-seed variation. Although the 5/5 success difference (20.7% vs. 6.2%) is significant under a simple binomial calculation, the average-length gap has no variance estimate, and the per-dimension test sizes in Table I (20 to 40 instructions) make the dimension-wise analysis in Fig. 7 statistically fragile. Please report the number of rollouts per condition, per-dimension success rates with intervals, and either seed variance or bootstrap confidence intervals for the headline gaps.
  4. [II-B and V-A] The claim of outperforming "state-of-the-art baselines" is not supported by the baseline set. Section II-B lists LCB, HiRT, RoboDual, DP-VLA, GR00T N1, and OpenHelix as related dual-system VLA models, but none of these appears in Tables II or III. The current baselines (3DDA, RoboFlamingo, LLaVA+3DDA, GPT-4o+3DDA) do not include a dual-system VLA from the cited related work. Please add at least one or two of these methods as baselines, or revise the claim to say that RationalVLA outperforms the evaluated baselines.
minor comments (5)
  1. [IV-C, Eq. (3)] Please define g_txt precisely, especially for defective and mixed instructions, and state explicitly whether L_CLIP is applied only to <ACT> tokens and not to <REJ> tokens.
  2. [V-C and Table III] The text says RationalVLA achieves "an average gain of 1.05 average length" over the strongest baseline, but Table III shows 2.48 - 1.42 = 1.06; please correct the number or explain the rounding.
  3. [Throughout] There are several typographical inconsistencies, most notably "LLaV A" instead of "LLaVA" in Sections V-A and V-B and in Table II; please standardize the notation.
  4. [VI-B and VI-C] The real-world results report success rates over 20 episodes without confidence intervals or trial-level variation; please state the sample size in the table captions and consider adding binomial confidence intervals.
  5. [Appendix A] The appendix states that GPT-4o-generated instructions could not satisfy the control-variable requirements for the visual, physical, semantic, and motion dimensions, so those dimensions rely on programmatic templates; please report how many unique templates and surface forms are used, since this affects the diversity and potential template overfitting of the benchmark.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; RationalVLA's central claims are empirical outcomes evaluated on a held-out test split and real-world trials, not derivations from the paper's own inputs.

full rationale

The paper's derivation chain is empirical rather than definitional. RationalVLA is trained on a combined CALVIN + RAMA training set (ratio 0.7:0.3, Section V) and evaluated on a disjoint 159-instruction RAMA test set and on CALVIN ABC to D, so the reported RAMA gains are not read off from training labels. The L_CLIP term in Eq. (3) uses ground-truth text descriptions gtxt only as auxiliary supervision to regularize the <ACT> latent embedding; it does not inject test-set outcomes or evaluation metrics into training. The paper contains self-citations in related work (e.g., OpenHelix [44] and prior quadruped VLA works by overlapping authors), but these are descriptive and not load-bearing: the dual-system architecture is also attributed to independent work such as LCB [38], HiRT [39], RoboDual [40], and GR00T N1 [43], and no claim is forced by a self-citation chain. The disclosed asymmetry that VLM-as-identifier baselines (LLaVA + 3DDA, GPT-4o + 3DDA) are not fine-tuned on RAMA because separate models cannot backpropagate (Section V-A) is a comparison-fairness confound, not circular reasoning, and the paper states it explicitly. Similarly, the protocol sentence in Section III-D that defective tasks are 'designed not to affect the proper initiation of subsequent tasks' is inconsistent with the failure case in Fig. 8, but this is an internal consistency issue rather than a self-justifying reduction. No equation is shown to be equivalent to its own input, no fitted parameter is renamed as a prediction, and no uniqueness theorem is imported from the authors' prior work.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The central claims rest on a stack of assumptions: that the MLLM can judge executability, the policy can be driven by latent embeddings, the generated defective instructions are genuinely infeasible, the evaluation protocol measures the right thing, and the CLIP alignment loss transfers semantics. None of these is independently verified outside the paper's own benchmark. There are no fitted physical constants; the free parameters are training hyperparameters.

free parameters (4)
  • Loss weights lambda_1, lambda_2, lambda_3 = 1, 100, 1
    Chosen by hand in Eq. 3; lambda_2 is 100 because the LLM needs extensive fine-tuning. These weights affect the balance of policy, language, and alignment objectives.
  • LoRA rank = 16
    Set for MLLM fine-tuning; no sensitivity analysis is reported.
  • Data mixture ratio (CALVIN:RAMA) = 0.7:0.3
    Chosen for the fine-tuning data mix; no ablation on this ratio is presented.
  • Training iterations = 5,000 stage 1; 30,000 stage 2
    Selected for the two-stage training; no ablation on training duration is reported.
assumptions (5)
  • domain assumption Fine-tuned LLaVA retains and sharpens multimodal reasoning needed to judge instruction executability from a single third-view image.
    The high-level MLLM must ground language against the scene; this is the core of the <REJ> mechanism and is only tested on the self-built RAMA benchmark.
  • domain assumption 3D Diffuser Actor can be conditioned on latent embeddings from <ACT> and generalize to unseen instructions at least as well as its native language conditioning.
    The policy interface replaces text S with latent z in Eq. 2; this requires that the learned embedding space carries sufficient task information.
  • domain assumption Defective instructions generated by programmatic part-of-speech replacement and GPT-4o prompts are consistently infeasible in the current scene.
    For example, 'orange block' is assumed non-existent when only a red block is present; any ambiguity in generation could mislabel executable instructions as defective.
  • domain assumption The CALVIN ABC-to-D evaluation with a defective prefix is a valid measure of robustness to defective instructions.
    The protocol is the paper's own design; there is no external benchmark for comparison, so the metric's validity rests on this assumption.
  • domain assumption The L_CLIP alignment loss using ground-truth text descriptions g_txt transfers the semantic content of the instruction into the latent embedding.
    This auxiliary loss is crucial (ablating it drops average length from 2.26 to 1.04), so the assumption that g_txt is a good training target is load-bearing.
invented entities (2)
  • <ACT> token
    purpose: Special vocabulary token whose last-layer hidden embedding is projected to condition the low-level policy to generate actions.
    Introduced by this paper; effectiveness is only demonstrated on the authors' own benchmark and real-world trials, with no external benchmark confirming its utility.
  • <REJ> token
    purpose: Special vocabulary token that, when projected, becomes an all-zero tensor to halt action execution for defective instructions.
    Same as above; the token's role is tied to the self-constructed RAMA benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RationalVLA: A Rational Vision-Language-Action Model with Dual System." pith.science (2026). https://pith.science/paper/CS4PPT3I

@misc{pith2026250610826,
  author       = {Pith},
  title        = {Pith review of: RationalVLA: A Rational Vision-Language-Action Model with Dual System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CS4PPT3I}},
  note         = {Machine review of arXiv:2506.10826}
}
read the original abstract

A fundamental requirement for real-world robotic deployment is the ability to understand and respond to natural language instructions. Existing language-conditioned manipulation tasks typically assume that instructions are perfectly aligned with the environment. This assumption limits robustness and generalization in realistic scenarios where instructions may be ambiguous, irrelevant, or infeasible. To address this problem, we introduce RAtional MAnipulation (RAMA), a new benchmark that challenges models with both unseen executable instructions and defective ones that should be rejected. In RAMA, we construct a dataset with over 14,000 samples, including diverse defective instructions spanning six dimensions: visual, physical, semantic, motion, safety, and out-of-context. We further propose the Rational Vision-Language-Action model (RationalVLA). It is a dual system for robotic arms that integrates the high-level vision-language model with the low-level manipulation policy by introducing learnable latent space embeddings. This design enables RationalVLA to reason over instructions, reject infeasible commands, and execute manipulation effectively. Experiments demonstrate that RationalVLA outperforms state-of-the-art baselines on RAMA by a 14.5% higher success rate and 0.94 average task length, while maintaining competitive performance on standard manipulation tasks. Real-world trials further validate its effectiveness and robustness in practical applications. Our project page is https://irpn-eai.github.io/RationalVLA.

Figures

Figures reproduced from arXiv: 2506.10826 by the authors.

Figure 1
Figure 1. Overview of two instructions. This figure illustrates two failure modes of traditional VLA models under unconventional instructions. (1) Complex and unseen instructions. Traditional manipulation policies and VLAs [3], [16] own limited language generalization abilities, thus executing wrong actions. (2) Defective instructions. Traditional VLA models have largely lost their language reasoning ability and tend to overf… view at source ↗
Figure 2
Figure 2. In the RAMA benchmark, we assess models across six dimensions: [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Experiment setting of CALVIN. CALVIN consists of four simulated [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 5
Figure 5. Figure 5: The comparison between different methods to tackle RAMA. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Overview of RationalVLA. Given the visual observation and text instruction that may be defective, an MLLM generates a text description of a task and an <ACT> token or a <REJ> token. The latent space embeddings from the <ACT> token’s last layer serve as a high-level lat…
Figure 7
Figure 7. Figure 7: Experimental results of our RationalVLA on the different dimensions [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Visualization of the rejection mechanism. The defective instructions are colored in [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Real-world robotic system setup. We employ a 6-DOF AgileX Piper arm equipped with a 1-DOF gripper and utilize a RealSense L515 camera to capture third-view images. the pepper in the bowl”, and “Stack green bowl on yellow bowl”, but their instructions are replaced with …
Figure 10
Figure 10. Figure 10: Representative results on defective instruction and unseen instruction of real-world experiments. The sequential images showcase that the robotic arm rejects the defective instructions, understands unseen instructions and completes the next task. REFERENCES [1] L. Ren…
Figure 11
Figure 11. Figure 11: Defective Instruction Generator for RAMA. Our defective instruction [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: Some examples of defective instructions in RAMA benchmark for Defective Instruction Generator. [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Prompt for analyzing the linguistic parts of instructions. [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]
Figure 14
Figure 14. Figure 14: Prompt for generating defective instructions of [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]
Figure 15
Figure 15. Figure 15: Chat templates for RationalVLA training process. [PITH_FULL_IMAGE:figures/full_fig_p014_15.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Token-Wise Latent Streaming from Slow Reasoners to Fast Planners for Dynamic Vision Language Navigation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Streaming intermediate hidden states from a slow VLM to a fast flow-matching planner, token by token, improves dynamic social VLN success and reduces observation staleness.

  2. Dual-Process Atomic Skill Learning: Decoupling Semantic Reasoning and Real-Time Control

    cs.RO 2026-07 conditional novelty 5.0 of 10

    Asynchronous dual-frequency hierarchical imitation learning with VQ skills and training-only latent diffusion improves compositional language-conditioned robot control and reduces skill codebook collapse.

  3. ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceiver

    cs.RO 2025-08 unverdicted novelty 4.0 of 10

    Adding a reconstruction target that redraws the object region makes a vision-language-action model focus its attention on the right object and manipulate more precisely.

Reference graph

Works this paper leans on

56 extracted references · 41 canonical work pages · cited by 3 Pith papers

  1. [1]

    Embodied intelligence toward future smart manufacturing in the era of ai foundation model,

    L. Ren, J. Dong, S. Liu, L. Zhang, and L. Wang, “Embodied intelligence toward future smart manufacturing in the era of ai foundation model,” IEEE/ASME Transactions on Mechatronics, pp. 1–11, 2024

    work page 2024

  2. [2]

    Rt-2: Vision-language-action models transfer web knowl- edge to robotic control,

    A. B. et al., “Rt-2: Vision-language-action models transfer web knowl- edge to robotic control,”Proceedings of The 7th Conference on Robot Learning (CoRL), 2023

    work page 2023

  3. [3]

    Vision-language foundation models as effective robot imitators,

    X. Li, M. Liu, H. Zhang, C. Yu, J. Xu, H. Wu, C. Cheang, Y . Jing, W. Zhang, H. Liu, H. Li, and T. Kong, “Vision-language foundation models as effective robot imitators,”The Twelfth International Confer- ence on Learning Representations (ICLR), 2024

    work page 2024

  4. [4]

    Quar-vla: Vision-language-action model for quadruped robots,

    P. Ding, H. Zhao, W. Zhang, W. Song, M. Zhang, S. Huang, N. Yang, and D. Wang, “Quar-vla: Vision-language-action model for quadruped robots,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 352–367

    work page 2024

  5. [5]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems, vol. 36, 2024

    2024

  6. [6]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radfordet al., “Gpt-4o system card,”arXiv preprint arXiv:2410.21276, 2024

    arXiv 2024

  7. [7]

    Lisa: Reasoning segmentation via large language model,

    X. Lai, Z. Tian, Y . Chen, Y . Li, Y . Yuan, S. Liu, and J. Jia, “Lisa: Reasoning segmentation via large language model,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 9579–9589

    work page 2024

  8. [8]

    Deepseek-vl: Towards real-world vision-language understanding,

    H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, H. Yanget al., “Deepseek-vl: Towards real-world vision-language understanding,”CoRR, 2024

    work page 2024

Show all 56 references
  1. [9]

    Cobra: Extending mamba to multi-modal large language model for efficient inference,

    H. Zhao, M. Zhang, W. Zhao, P. Ding, S. Huang, and D. Wang, “Cobra: Extending mamba to multi-modal large language model for efficient inference,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 10, 2025, pp. 10 421–10 429

    2025

  2. [10]

    Seeing far and clearly: Mitigating hallucina- tions in mllms with attention causal decoding,

    F. Tang, C. Liu, Z. Xu, M. Hu, Z. Huang, H. Xue, Z. Chen, Z. Peng, Z. Yang, S. Zhouet al., “Seeing far and clearly: Mitigating hallucina- tions in mllms with attention causal decoding,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 26 147– 26 159

    2025

  3. [11]

    Rt-1: 11 Robotics transformer for real-world control at scale,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsuet al., “Rt-1: 11 Robotics transformer for real-world control at scale,”Robotics: Science and Systems, 2023

    2023

  4. [12]

    Germ: A generalist robotic model with mixture-of-experts for quadruped robot,

    W. Song, H. Zhao, P. Ding, C. Cui, S. Lyu, Y . Fan, and D. Wang, “Germ: A generalist robotic model with mixture-of-experts for quadruped robot,” in2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 11 879–11 886

    2024

  5. [13]

    Octo: An open- source generalist robot policy,

    O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, J. Luo, Y . L. Tan, P. R. Sanketi, Q. Vuong, T. Xiao, D. Sadigh, C. Finn, and S. Levine, “Octo: An open- source generalist robot policy,”Robotics: Science and Systems, 2024

    2024

  6. [14]

    More: Unlocking scalability in reinforcement learning for quadruped vision-language-action models,

    H. Zhao, W. Song, D. Wang, X. Tong, P. Ding, X. Cheng, and Z. Ge, “More: Unlocking scalability in reinforcement learning for quadruped vision-language-action models,”arXiv preprint arXiv:2503.08007, 2025

    2025 arXiv

  7. [15]

    Accelerating vision-language-action model inte- grated with action chunking via parallel decoding,

    W. Song, J. Chen, P. Ding, H. Zhao, W. Zhao, Z. Zhong, Z. Ge, J. Ma, and H. Li, “Accelerating vision-language-action model inte- grated with action chunking via parallel decoding,”arXiv preprint arXiv:2503.02310, 2025

    2025

  8. [16]

    3d diffuser actor: Policy diffusion with 3d scene representations,

    T.-W. Ke, N. Gkanatsios, and K. Fragkiadaki, “3d diffuser actor: Policy diffusion with 3d scene representations,” in8th Annual Conference on Robot Learning

  9. [17]

    Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks,

    O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard, “Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks,”IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 7327–7334, 2022

    2022

  10. [18]

    Vision-language- action model with open-world embodied reasoning from pretrained knowledge,

    Z. Zhou, Y . Zhu, J. Wen, C. Shen, and Y . Xu, “Vision-language- action model with open-world embodied reasoning from pretrained knowledge,”arXiv preprint arXiv:2505.21906, 2025

    2025 arXiv

  11. [19]

    Dynamic neural networks: A survey,

    Y . Han, G. Huang, S. Song, L. Yang, H. Wang, and Y . Wang, “Dynamic neural networks: A survey,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, pp. 7436–7456, 2021

    2021

  12. [20]

    Gsva: Generalized segmentation via multimodal large language models,

    Z. Xia, D. Han, Y . Han, X. Pan, S. Song, and G. Huang, “Gsva: Generalized segmentation via multimodal large language models,”2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3858–3869, 2023

    2024

  13. [21]

    A multimodal robust recognition method for grasping objects with robot flexible grippers,

    Q. Liang, W. Xiao, J. Long, and D. Zhang, “A multimodal robust recognition method for grasping objects with robot flexible grippers,” IEEE/ASME Transactions on Mechatronics, vol. 30, no. 2, pp. 1154– 1165, 2025

    2025

  14. [22]

    Learning generalizable vision-tactile robotic grasping strategy for deformable objects via transformer,

    Y . Han, K. Yu, R. Batra, N. Boyd, C. Mehta, T. Zhao, Y . She, S. Hutchin- son, and Y . Zhao, “Learning generalizable vision-tactile robotic grasping strategy for deformable objects via transformer,”IEEE/ASME Transac- tions on Mechatronics, vol. 30, no. 1, pp. 554–566, 2025

    2025

  15. [23]

    Dih-tele: Dexterous in-hand teleoperation framework for learning mul- tiobjects manipulation with tactile sensing,

    J. Huang, K. Chen, J. Zhou, X. Lin, P. Abbeel, Q. Dou, and Y . Liu, “Dih-tele: Dexterous in-hand teleoperation framework for learning mul- tiobjects manipulation with tactile sensing,”IEEE/ASME Transactions on Mechatronics, pp. 1–12, 2025

    2025

  16. [24]

    Efficient grasp detection network with gaussian-based grasp representation for robotic manipulation,

    H. Cao, G. Chen, Z. Li, Q. Feng, J. Lin, and A. Knoll, “Efficient grasp detection network with gaussian-based grasp representation for robotic manipulation,”IEEE/ASME Transactions on Mechatronics, vol. 28, no. 3, pp. 1384–1394, 2023

    2023

  17. [25]

    Language conditioned imitation learning over unstructured data,

    C. Lynch and P. Sermanet, “Language conditioned imitation learning over unstructured data,”Robotics: Science and Systems, 2021

    2021

  18. [26]

    What matters in language conditioned robotic imitation learning over unstructured data,

    O. Mees, L. Hermann, and W. Burgard, “What matters in language conditioned robotic imitation learning over unstructured data,”IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 11 205–11 212, 2022

    2022

  19. [27]

    Diffusion policy: Visuomotor policy learning via ac- tion diffusion,

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song, “Diffusion policy: Visuomotor policy learning via ac- tion diffusion,”The International Journal of Robotics Research, p. 02783649241273668, 2023

    2023

  20. [28]

    3d diffusion policy,

    Y . Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu, “3d diffusion policy,”Robotics: Science and Systems, 2024

    2024

  21. [29]

    Consistency policy: Accelerated visuomotor policies via consistency distillation,

    A. Prasad, K. Lin, J. Wu, L. Zhou, and J. Bohg, “Consistency policy: Accelerated visuomotor policies via consistency distillation,”Robotics: Science and Systems, 2024

    2024

  22. [30]

    RT-trajectory: Robotic task generalization via hindsight trajectory sketches,

    J. Gu, S. Kirmani, P. Wohlhart, Y . Lu, M. G. Arenas, K. Rao, W. Yu, C. Fu, K. Gopalakrishnan, Z. Xu, P. Sundaresan, P. Xu, H. Su, K. Hausman, C. Finn, Q. Vuong, and T. Xiao, “RT-trajectory: Robotic task generalization via hindsight trajectory sketches,” inThe Twelfth Internat...

    2024

  23. [31]

    Sara-rt: Scaling up robotics transformers with self-adaptive robust attention,

    I. Leal, K. Choromanski, D. Jain, A. Dubey, J. Varley, M. Ryoo, Y . Lu, F. Liu, V . Sindhwani, Q. Vuonget al., “Sara-rt: Scaling up robotics transformers with self-adaptive robust attention,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, p...

    2024

  24. [32]

    Inner monologue: Em- bodied reasoning through planning with language models,

    W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y . Chebotaret al., “Inner monologue: Em- bodied reasoning through planning with language models,”Conference on Robot Learning (CoRL), 2023

    2023

  25. [33]

    Unleashing large-scale video generative pre-training for visual robot manipulation,

    H. Wu, Y . Jing, C. Cheang, G. Chen, J. Xu, X. Li, M. Liu, H. Li, and T. Kong, “Unleashing large-scale video generative pre-training for visual robot manipulation,” inThe Twelfth International Conference on Learning Representations

  26. [34]

    Video language planning,

    Y . Du, S. Yang, P. Florence, F. Xia, A. Wahid, P. Sermanet, T. Yu, P. Abbeel, J. B. Tenenbaum, L. P. Kaelblinget al., “Video language planning,” inThe Twelfth International Conference on Learning Repre- sentations

  27. [35]

    Robomamba: Efficient vision-language-action model for robotic reasoning and manipulation,

    J. Liu, M. Liu, Z. Wang, P. An, X. Li, K. Zhou, S. Yang, R. Zhang, Y . Guo, and S. Zhang, “Robomamba: Efficient vision-language-action model for robotic reasoning and manipulation,” inNeural Information Processing Systems, 2024

    2024

  28. [36]

    Vlas: Vision-language-action model with speech instructions for customized robot manipulation,

    W. Zhao, P. Ding, Z. Min, Z. Gong, S. Bai, H. Zhao, and D. Wang, “Vlas: Vision-language-action model with speech instructions for customized robot manipulation,” inThe Thirteenth International Conference on Learning Representations

  29. [37]

    Dual-arm robotic fabric manipulation with quasi-static and dynamic primitives for rapid garment flattening,

    C. Zhou, R. Jiang, F. Luan, S. Meng, Z. Wang, Y . Dong, Y . Zhou, and B. He, “Dual-arm robotic fabric manipulation with quasi-static and dynamic primitives for rapid garment flattening,”IEEE/ASME Transactions on Mechatronics, pp. 1–11, 2025

    2025

  30. [38]

    From llms to actions: Latent codes as bridges in hierarchical robot control,

    Y . Shentu, P. Wu, A. Rajeswaran, and P. Abbeel, “From llms to actions: Latent codes as bridges in hierarchical robot control,”IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024

    2024

  31. [39]

    Hirt: Enhancing robotic control with hierarchical robot transformers,

    J. Zhang, Y . Guo, X. Chen, Y .-J. Wang, Y . Hu, C. Shi, and J. Chen, “Hirt: Enhancing robotic control with hierarchical robot transformers,” in8th Annual Conference on Robot Learning

  32. [40]

    Towards synergistic, generalized, and efficient dual-system for robotic manipulation,

    Q. Bu, H. Li, L. Chen, J. Cai, J. Zeng, H. Cui, M. Yao, and Y . Qiao, “Towards synergistic, generalized, and efficient dual-system for robotic manipulation,” 2025

    2025

  33. [41]

    A dual process vla: Efficient robotic manipulation leveraging vlm,

    B. Han, J. Kim, and J. Jang, “A dual process vla: Efficient robotic manipulation leveraging vlm,”arXiv preprint arXiv:2410.15549, 2024

    2024 arXiv

  34. [42]

    Openvla: An open-source vision-language-action model,

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuonget al., “Openvla: An open-source vision-language-action model,” in8th Annual Conference on Robot Learning

  35. [43]

    Gr00t n1: An open foundation model for generalist humanoid robots,

    J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, Y . Fang, D. Fox, F. Hu, S. Huang, J. Janget al., “Gr00t n1: An open foundation model for generalist humanoid robots,”Arxiv, 2025

    2025

  36. [44]

    Openhelix: A short survey, empirical analysis, and open-source dual-system vla model for robotic manipulation,

    C. Cui, P. Ding, W. Song, S. Bai, X. Tong, Z. Ge, R. Suo, W. Zhou, Y . Liu, B. Jiaet al., “Openhelix: A short survey, empirical analysis, and open-source dual-system vla model for robotic manipulation,”arXiv preprint arXiv:2505.03912, 2025

    2025 arXiv

  37. [45]

    Hierarchical reinforcement learning with model guidance for mobile manipulation,

    Y . Zhou, Y . Zhou, K. Jin, and H. Wang, “Hierarchical reinforcement learning with model guidance for mobile manipulation,”IEEE/ASME Transactions on Mechatronics, pp. 1–9, 2025

    2025

  38. [46]

    Interactive imitation learning of bimanual movement primitives,

    G. Franzese, L. d. S. Rosa, T. Verburg, L. Peternel, and J. Kober, “Interactive imitation learning of bimanual movement primitives,” IEEE/ASME Transactions on Mechatronics, vol. 29, no. 5, pp. 4006– 4018, 2024

    2024

  39. [47]

    Navigating beyond in- structions: Vision-and-language navigation in obstructed environments,

    H. Hong, S. Wang, Z. Huang, Q. Wu, and J. Liu, “Navigating beyond in- structions: Vision-and-language navigation in obstructed environments,” inProceedings of the 32nd ACM International Conference on Multime- dia, 2024, pp. 7639–7648

    2024

  40. [48]

    Badnaver: Exploring jail- break attacks on vision-and-language navigation,

    W. Lyu, Z. Li, Y . Qiao, and Q. Wu, “Badnaver: Exploring jail- break attacks on vision-and-language navigation,”arXiv preprint arXiv:2505.12443, 2025

    2025 arXiv

  41. [49]

    Safety bounds in human robot interaction: A survey,

    A. Zacharaki, I. Kostavelis, A. Gasteratos, and I. Dokas, “Safety bounds in human robot interaction: A survey,”Safety science, vol. 127, p. 104667, 2020

    2020

  42. [50]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” 2023

    2023

  43. [51]

    Pybullet, a python module for physics simulation for games, robotics and machine learning,

    E. Coumans and Y . Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” http://pybullet.org, 2016–2019

    2016

  44. [52]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inInternational Conference on Learning Representations (ICLR), 2022

    2022

  45. [53]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PMLR, 2021, pp. 8748–8763

    2021

  46. [54]

    Investigating the catastrophic forgetting in multimodal large language models,

    Y . Zhai, S. Tong, X. Li, M. Cai, Q. Qu, Y . J. Lee, and Y . Ma, “Investigating the catastrophic forgetting in multimodal large language models,”NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following., 2023. 12

    2023

  47. [55]

    Allava: Harnessing gpt4v-synthesized data for lite vision-language models,

    G. H. Chen, S. Chen, R. Zhang, J. Chen, X. Wu, Z. Zhang, Z. Chen, J. Li, X. Wan, and B. Wang, “Allava: Harnessing gpt4v-synthesized data for lite vision-language models,” 2024

    2024

  48. [56]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” 2023. APPENDIX We sincerely thank Yuxin Huang for cre...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.