Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

This paper claims that robot control generalizes best when the VLM only outputs sparse 3D waypoints and a separate, frozen action expert turns them, together with point clouds, into dense executable actions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 11:33 UTC pith:BXDLH6SB

load-bearing objection Clean modular VLA design with genuinely held-out evidence on ManiSkill and real hardware, but the RoboTwin numbers likely leak pretraining data—worth review with a clear demand for a train/test split. the 4 major comments →

arxiv 2510.03896 v2 pith:BXDLH6SB submitted 2025-10-04 cs.CV cs.RO

GAE: Unleashing Physical Potential of VLM with Generalizable Action Expert

Mingyu Liu , Zheng Huang , Xiaoyi Lin , Muzhi Zhu , Canyu Zhao , Yating Wang , Haoyi Zhu , Hao Chen
show 1 more author
Chunhua Shen
This is my paper
classification cs.CV cs.RO
keywords vision-language-action modelsrobot manipulationaction expertsparse 3D waypointsdiffusion policypoint cloudzero-shot generalizationaction pretraining
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that vision-language-action models generalize poorly because they force the action module to interpret semantic information, and that this can be fixed by a clean geometric interface. The proposed framework uses a VLM to predict sparse 3D waypoints in the camera frame, then a separate Generalizable Action Expert refines those waypoints into dense action trajectories using real-time point-cloud observations. To train that expert, the paper introduces an Action Pre-training, Pointcloud Fine-tuning scheme: first learn trajectory following on large batches of pure trajectory data, then learn environment-aware refinement. If the approach works as claimed, a frozen action expert can be reused zero-shot across new tasks, camera viewpoints, and language instructions, with only lightweight fine-tuning of the VLM.

Core claim

The central claim is that a task-agnostic action expert can become genuinely generalizable if it is freed from semantic interpretation and given an explicit, sparse geometric interface. GAE is a conditional diffusion policy conditioned on robot proprioceptive state, a guidance pose sampled from a B-spline interpolation of VLM-predicted sparse 3D waypoints, and a cropped point cloud of the local environment. During action pretraining the point-cloud conditioning is masked so the model learns pure trajectory-following at very large batch scale; during point-cloud fine-tuning it learns to refine trajectories using local geometry. After training, GAE is frozen and reused across downstream tasks,

What carries the argument

The load-bearing mechanism is the sparse 3D waypoint interface: the VLM outputs a small set of 3D waypoints and a final gripper pose in the camera frame, a B-spline interpolates them into a continuous trajectory, and the action expert refines this trajectory with point-cloud observations. This explicit geometric interface—rather than semantic embeddings, visual features, or attention maps—is what decouples planning from execution and makes large-scale pretraining of the action expert feasible.

Load-bearing premise

The paper's zero-shot claim depends on the main evaluation tasks not having been part of the pretraining trajectories, since the data appendix lists the same simulator used for evaluation as one of the pretraining sources.

What would settle it

Check the pretraining trajectory list against the evaluation task configurations in the main results: if any of the evaluated tasks (same scenes, objects, and initial conditions) appear among the roughly 8,000 trajectories from that simulator used to train the action expert, the zero-shot numbers are confounded. Alternatively, rerun the evaluation on a suite of tasks from the same simulator that was deliberately held out of pretraining; if success drops materially, the claimed generalization is partly memorization.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, long-horizon manipulation becomes the framework's strength: on three long-horizon evaluation tasks, the method reports about 60% average success while single-task expert baselines land near zero.
  • The VLM can be fine-tuned for far fewer steps before performance saturates, preserving its language and reasoning abilities instead of sacrificing them to action tuning.
  • The same frozen action expert can be reused across multiple downstream tasks and real-world settings, with only the VLM updated, lowering deployment cost.
  • Because action pretraining and point-cloud fine-tuning are separated, the pretraining stage can use very large batch sizes, making large-scale trajectory learning substantially more efficient.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the geometric interface is the true source of generalization, then swapping in any VLM that can reliably produce camera-frame 3D waypoints should preserve most of the action expert's transfer; the paper does not isolate this modularity claim directly.
  • The same design may transfer to other robot embodiments by canonicalizing point clouds and actions into a shared frame, but the paper only demonstrates a single arm, so cross-embodiment generalization remains an untested extension.
  • The zero-shot claim would be strongest if the evaluation tasks were explicitly held out from pretraining; the paper's data appendix lists the evaluation simulator as a pretraining source, so a reader should verify the exact trajectory split before taking the benchmark numbers as pure generalization.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes GAE, a two-module framework that decouples high-level planning from low-level control. A VLM predicts sparse 3D waypoints and a target gripper pose in the camera frame; after spline interpolation and coordinate-frame transformation, a frozen diffusion-based 'action expert' consumes these waypoints together with real-time point-cloud observations and robot proprioception to generate dense SE(3) actions. The action expert is trained with an 'Action Pre-training, Point-cloud Fine-tuning' (APPF) scheme on 150k trajectories from seven simulators and real-world datasets. Experiments are reported on RoboTwin, ManiSkill, real-world Franka tasks, and several generalization-oriented ablations. The central claim is that the action expert, once pretrained, can be deployed zero-shot across tasks and environments, requiring only lightweight VLM fine-tuning to produce the sparse geometric interface.

Significance. If the robustness and generalization claims hold, the paper makes a useful contribution: the sparse 3D-waypoint interface is clean, it removes semantic interpretation from the low-level policy, and the APPF training scheme is a plausible way to scale action pretraining. The ManiSkill results (Section 4.3, Table 5) are encouraging because that environment is explicitly excluded from pretraining, and the real-world tasks are new to the action expert. The viewpoint generalization results (Table 3) also provide some support for the method's robustness. However, the headline quantitative evidence on the RoboTwin benchmark is currently not clean evidence for zero-shot generalization, because RoboTwin is also a pretraining source and no train/test split is given. The paper's strongest abstract claim, 'surpasses generalist models on all tasks,' rests on this potentially contaminated comparison.

major comments (4)
  1. [§3.2.1, §4.2, Appendix A.1, Table 10 vs. Table 1] RoboTwin is listed as both a pretraining data source and the main evaluation benchmark. Section 3.2.1 states that trajectories were replayed from multiple simulators 'including RoboTwin,' and Appendix A.1/Table 10 report 8,000 RoboTwin trajectories. The paper never states that the 11 evaluation tasks in Table 1 were excluded from these 8,000 trajectories, whereas Section 4.3 explicitly notes that ManiSkill was 'entirely excluded from pre-training.' Without an explicit train/test split, the Table 1 numbers may reflect in-domain fitting rather than zero-shot generalization. This is load-bearing for the abstract claim of surpassing generalist models on all tasks.
  2. [§4.5.2, Table 8] The noise-scale hyperparameter is selected based on performance on the same RoboTwin tasks used in Table 1. The text says the optimal scale of 0.1 was chosen 'achieving optimal performance on the tasks in Table 1.' Since the action expert is otherwise frozen, this means the main benchmark results are obtained with a hyperparameter tuned on the evaluation set. The long-horizon success rate changes from 0.43 to 0.60 depending on this choice. To support zero-shot generalization, the noise scale should be selected on held-out tasks or its sensitivity should be reported on tasks excluded from tuning.
  3. [§4.4, Table 6, Abstract] The phrase 'zero-shot deployment' is used in the abstract and introduction without sufficient qualification. In the real-world experiments, the VLM is fine-tuned on 300 task-specific demonstrations, and only the action expert is frozen. The paper's own Section 4.1 is clearer, stating that the action expert is deployed zero-shot while the VLM undergoes SFT. The broader wording overstates what is demonstrated. Please consistently state that zero-shot refers to the action expert only, not the full VLM+expert system.
  4. [§4.3, Table 5 and §4.2, Table 1] The independent generalization evidence is thinner than the paper's claims suggest. ManiSkill covers only three tasks, and the real-world tasks are designed by the authors, so they support the claim qualitatively but do not establish broad coverage across environments and object categories. This is not a fatal issue by itself, but the conclusion should be calibrated to the amount of held-out evidence available, especially because the larger RoboTwin benchmark may be contaminated.
minor comments (5)
  1. [§4.5.1, Figure 5] Typographical issues: 'demostrated' appears in Section 4.5.1, and Figure 5's caption contains 'We are here!' which appears to be a leftover annotation. Figure 1 also has 'Robot Sapce' and 'DirectlyExcute' typos.
  2. [Table 1] The asterisks on Pi0* and RDT* are never defined in the table caption or text. Also, the number of evaluation episodes and random seeds is not reported, so no confidence intervals or statistical significance can be assessed.
  3. [§3.2.1 and Appendix A.1] The text refers to 'RoboTwin' in Section 3.2.1 but Appendix A.1 describes 'RoboTwin 2.0.' Please clarify the exact version and, more importantly, whether the benchmark evaluation tasks are a subset of the pretraining trajectories.
  4. [§1, Contributions] The contribution bullet says 'To the best of our knowledge, this is the first attempt to train a generalizable expert that can be deployed without task-specific fine-tuning.' This is difficult to verify and should either be softened or supported with a more explicit comparison to prior dual-system and generalist-policy work.
  5. [Table 2] The 'Training Steps (All tasks training)' row for the proposed method only reports 8*32*1000, which appears to cover the VLM SFT steps. Please clarify whether this row represents only the VLM phase and where the action expert's compute budget is reported.

Circularity Check

0 steps flagged

No significant circularity found; the central generalization claim rests on independent evaluations rather than on a construction that folds its inputs into its outputs.

full rationale

GAE's contribution is an empirical architecture: the VLM predicts sparse 3D waypoints (Sec. 3.1), and the action expert is trained as a conditional diffusion policy (Eqs. 3-4). There is no equation in which a fitted parameter is renamed as a prediction or in which a benchmark result is the training objective by construction. The noise-scale ablation (Sec. 4.5.2) selects a hyperparameter using the Table 1 tasks; this is a test-set-selection / evaluation-validity concern, not a circular derivation, because the resulting success rates are not equal to the training loss or to the selected hyperparameter by construction. Self-citations are not load-bearing: the architecture is credited to external work (3D Diffusion Policy, Ze et al. 2024), and the benchmark citations (RoboTwin, ManiSkill) are external datasets, not prior results by the current authors invoked to force the design. The main generalization claim is independently supported by the ManiSkill evaluation, which the paper states was "entirely excluded from pre-training" (Sec. 4.3), and by real-world deployment with the action expert frozen (Sec. 4.4). The only substantive concern is that RoboTwin appears both as a pretraining source (Table 10: 8,000 trajectories) and as the main benchmark (Table 1), with no explicit train/test split stated for those evaluation tasks. That is a potential data-overlap / zero-shot-claim validity issue, but it is not evidence of equivalence-by-construction: the ManiSkill and real-world results remain independent of the RoboTwin pretraining distribution. Honest finding: no significant circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The paper's contribution rests on a set of empirical assumptions about data, interface sufficiency, and sensor quality. The main free parameters are the goal-pose noise scale and the VLM SFT step count; both are selected on the evaluation benchmarks. No new physical entities are introduced.

free parameters (2)
  • goal pose noise scale = 0.1
    Added to VLM-generated goal poses during expert training; Table 8 ablates and selects 0.1 on the RoboTwin benchmark, which is also the main evaluation set.
  • VLM SFT steps = 1000
    Number of supervised fine-tuning steps for the VLM; 1000 is used in main experiments (Table 2/A.4), chosen as a trade-off in Figure 4/Table 7.
axioms (5)
  • domain assumption The evaluation tasks in RoboTwin (Table 1) are not part of the action expert's pretraining distribution
    The paper never states whether the 8,000 RoboTwin pretraining trajectories include the 11 evaluation tasks; the zero-shot generalization claim depends on this.
  • domain assumption Sparse 3D waypoints form a sufficient interface for the tested manipulation skills
    The method asserts that waypoints plus a target gripper pose are enough for the expert to execute pick/place/stack/drawer tasks; contact-rich or deformable-object tasks may need richer guidance.
  • domain assumption Point clouds at inference time are of sufficient quality for the expert
    The expert conditions on cropped point clouds; in real-world deployment the camera must provide reliable depth, and the paper uses estimated depth from stereo or monocular models in training.
  • domain assumption Cross-embodiment pretraining transfers to the Franka Research 3 arm
    The pretraining data include WidowX, AgiBot, and other arms; the real-world evaluation is only on Franka, and no cross-embodiment evaluation is shown.
  • ad hoc to paper Action pre-training without point clouds, followed by point-cloud fine-tuning, does not cause catastrophic forgetting of motion skills
    The APPF scheme is proposed without a formal analysis; the paper provides empirical convergence curves but no proof that the point-cloud fine-tuning stage preserves the pretrained trajectory-following skill.

pith-pipeline@v1.3.0-alltime-deepseek · 15442 in / 12187 out tokens · 91659 ms · 2026-08-04T11:33:24.698841+00:00 · methodology

0 comments
read the original abstract

Vision-language models demonstrate strong reasoning and planning abilities, yet grounding these predictions into precise robot actions remains a central challenge. Existing Vision-Language-Action methods typically entangle reasoning and action generation, leading to limited generalization. We propose Generalizable Action Expert (GAE), a task-agnostic model that converts sparse geometric plans into dense robot actions. Our approach introduces a sparse geometric interface: the VLM predicts sparse 3D waypoints representing high-level intention, while GAE maps these waypoints together with real-time point cloud observations to continuous action trajectories. GAE is pretrained on a large-scale pointcloud-trajectory dataset comprising 150k trajectories from both simulation and real-world robots. To further improve efficiency and generalization, we introduce an Action Pre-training, Pointcloud Fine-tuning (APPF) scheme that decouples learning action dynamics from geometry grounding. After pretraining, GAE is frozen and reused across downstream tasks, requiring only lightweight fine-tuning of the VLM to produce the sparse interface. Experiments show that our method achieves strong performance and generalization across diverse visual domains, camera viewpoints, and natural language instructions.

Figures

Figures reproduced from arXiv: 2510.03896 by Canyu Zhao, Chunhua Shen, Hao Chen, Haoyi Zhu, Mingyu Liu, Muzhi Zhu, Xiaoyi Lin, Yating Wang, Zheng Huang.

Figure 1
Figure 1. Figure 1: The pipeline of our proposed method. Our approach begins with a VLM predicting a sparse set of 3D waypoints directly in the camera frame, preserving its vision-centric knowledge. These sparse points are then transformed and interpolated via a B-spline into a continuous and smooth pose trajectory to provide dense guidance for a low-level action expert. keyframes for annotation at moments where the gripper’s… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our data annotation pipeline. We construct our SFT dataset by first select￾ing keyframes based on gripper state changes. 3.2.1 DATA PREPARATION Training a well-generalized action expert model requires a large volume of high-quality 3D point cloud annotations from both real-world and simulated environments. In simulation, obtaining high-fidelity point clouds is straightforward due to precise dep… view at source ↗
Figure 3
Figure 3. Figure 3: Real World Task Setting [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ablation on training strategy [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Simulation Robot Results 16 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Real World Robot Results 17 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Perfect Demo Makes Poor Teacher: Learning Robust Alignment from Critical Motion Segments

    cs.RO 2026-06 conditional novelty 6.0

    Fluent expert demonstrations under-supervise the short alignment phase that decides success, and a compact spatio-temporal dynamic feature (STAIR) recovers most of the deliberate-demonstration gain from fluent data alone.

Reference graph

Works this paper leans on

55 extracted references · 29 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems

    AgiBot-World-Contributors, Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang, Shu Jiang, Yuxin Jiang, Cheng Jing, Hongyang Li, Jialu Li, Chiming Liu, Yi Liu, Yuxiang Lu, Jianlan Luo, Ping Luo, Yao Mu, Yuehan Niu, Yixuan Pan, Jiangmiao Pang, Yu Qiao, Guanghui Ren, Cheng Ruan, Jiaqi Shan, Yongjian Shen, Ch...

    Pith/arXiv arXiv 2025

  2. [2]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 19129--19139, 2022

    2022

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Gr00t n1: An open foundation model for generalist humanoid robots

    Johan Bjorck, Fernando Casta \ n eda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    0: A vision-language-action flow model for general robot control

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. 0: A vision-language-action flow model for general robot control. corr, abs/2410.24164, 2024. doi: 10.48550. arXiv preprint ARXIV.2410.24164

    Pith/arXiv arXiv 2024

  6. [6]

    Rt-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022

    Pith/arXiv arXiv 2022

  7. [7]

    Spatialbot: Precise spatial understanding with vision language models

    Wenxiao Cai, Iaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. arXiv preprint arXiv:2406.13642, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation

    Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, et al. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 14455--14465, 2024 a

    2024

  10. [10]

    Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation

    Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Qiwei Liang, Zixuan Li, Xianliang Lin, Yiheng Ge, Zhenyu Gu, et al. Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 24185--24198, 2024 b

    2024

  12. [12]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, pp.\ 02783649241273668, 2023

    2023

  13. [13]

    Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots

    Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. arXiv preprint arXiv:2402.10329, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Scaffolding dexterous manipulation with vision-language models

    Vincent de Bakker, Joey Hejna, Tyler Ga Wei Lum, Onur Celik, Aleksandar Taranovic, Denis Blessing, Gerhard Neumann, Jeannette Bohg, and Dorsa Sadigh. Scaffolding dexterous manipulation with vision-language models. arXiv preprint arXiv:2506.19212, 2025

    arXiv 2025

  16. [16]

    Self-supervised 6d object pose estimation for robot manipulation

    Xinke Deng, Yu Xiang, Arsalan Mousavian, Clemens Eppner, Timothy Bretl, and Dieter Fox. Self-supervised 6d object pose estimation for robot manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp.\ 3665--3671. IEEE, 2020

    2020

  17. [17]

    Scaling up and distilling down: Language-guided robot skill acquisition

    Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Language-guided robot skill acquisition. In Conference on Robot Learning, pp.\ 3766--3777. PMLR, 2023

    2023

  18. [18]

    Thinkact: Vision-language-action reasoning via reinforced visual latent planning

    Chi-Pin Huang, Yueh-Hua Wu, Min-Hung Chen, Yu-Chiang Frank Wang, and Fu-En Yang. Thinkact: Vision-language-action reasoning via reinforced visual latent planning. arXiv preprint arXiv: 2507.16815, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch...

    Pith/arXiv arXiv 2025

  20. [20]

    Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark & learning environment. arXiv preprint arXiv: 1909.12271, 2019

    Pith/arXiv arXiv 1909

  21. [21]

    Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, You...

    Pith/arXiv arXiv 2025

  22. [22]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Bridgevla: Input-output alignment for efficient 3d manipulation learning with vision-language models

    Peiyan Li, Yixiang Chen, Hongtao Wu, Xiao Ma, Xiangnan Wu, Yan Huang, Liang Wang, Tao Kong, and Tieniu Tan. Bridgevla: Input-output alignment for efficient 3d manipulation learning with vision-language models. arXiv preprint arXiv: 2506.07961, 2025

    arXiv 2025

  24. [24]

    HAMSTER : Hierarchical action models for open-world robot manipulation

    Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Caelan Reed Garrett, Fabio Ramos, Dieter Fox, Anqi Li, Abhishek Gupta, and Ankit Goyal. HAMSTER : Hierarchical action models for open-world robot manipulation. In 1st Workshop on X-Embodiment Robot Learning, 2024. URL https://openreview.net/forum?id=yF3UekSJus

    2024

  25. [25]

    Prompting depth anything for 4k resolution accurate metric depth estimation

    Haotong Lin, Sida Peng, Jingxiao Chen, Songyou Peng, Jiaming Sun, Minghuan Liu, Hujun Bao, Jiashi Feng, Xiaowei Zhou, and Bingyi Kang. Prompting depth anything for 4k resolution accurate metric depth estimation. Cvpr, 2025

    2025

  26. [26]

    Libero: Benchmarking knowledge transfer for lifelong robot learning

    Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36: 0 44776--44791, 2023

    2023

  27. [27]

    Sqa3d: Situated question answering in 3d scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022

    Pith/arXiv arXiv 2022

  28. [28]

    kpam: Keypoint affordances for category-level robotic manipulation

    Lucas Manuelli, Wei Gao, Peter Florence, and Russ Tedrake. kpam: Keypoint affordances for category-level robotic manipulation. In The International Symposium of Robotics Research, pp.\ 132--157. Springer, 2019

    2019

  29. [29]

    Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks

    Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7 0 (3): 0 7327--7334, 2022

    2022

  30. [30]

    Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations

    Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. arXiv preprint arXiv:2107.14483, 2021

    Pith/arXiv arXiv 2021

  31. [31]

    Robotwin: Dual-arm robot benchmark with generative digital twins

    Yao Mu, Tianxing Chen, Zanxin Chen, Shijia Peng, Zhiqian Lan, Zeyu Gao, Zhixuan Liang, Qiaojun Yu, Yude Zou, Mingkun Xu, et al. Robotwin: Dual-arm robot benchmark with generative digital twins. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 27649--27660, 2025

    2025

  32. [32]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R \"a dle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics

    Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 15768--15780, 2025

    2025

  34. [34]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Vuong, Andre Wang He, Vivek Myers, Kuan Fang, Chelsea Finn, and S

    Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Maximilian Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Q. Vuong, Andre Wang He, Vivek Myers, Kuan Fang, Chelsea Finn, and S. Levine. Bridgedata v2: A dataset for robot learning at scale. Conference on Robot Learning, 2023. doi:10.48550/arXiv.2308.12952

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.12952 2023

  36. [36]

    Moge: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision

    Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. Moge: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision. Computer Vision and Pattern Recognition, 2024. doi:10.1109/CVPR52734.2025.00496

    arXiv 2024

  37. [37]

    Moge-2: Accurate monocular geometry with metric scale and sharp details

    Ruicheng Wang, Sicheng Xu, Yue Dong, Yu Deng, Jianfeng Xiang, Zelong Lv, Guangzhong Sun, Xin Tong, and Jiaolong Yang. Moge-2: Accurate monocular geometry with metric scale and sharp details. arXiv preprint arXiv: 2507.02546, 2025 a

    Pith/arXiv arXiv 2025

  38. [38]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025 b

    Pith/arXiv arXiv 2025

  39. [39]

    Foundationstereo: Zero-shot stereo matching

    Bowen Wen, Matthew Trepte, Joseph Aribido, Jan Kautz, Orazio Gallo, and Stan Birchfield. Foundationstereo: Zero-shot stereo matching. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 5249--5260, 2025 a

    2025

  40. [40]

    Tinyvla: Towards fast, data-efficient vision-language-action models for robotic manipulation

    Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Zhibin Tang, Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, et al. Tinyvla: Towards fast, data-efficient vision-language-action models for robotic manipulation. IEEE Robotics and Automation Letters, 2025 b

    2025

  41. [41]

    Spatial-mllm: Boosting mllm capabilities in visual-based spatial intelligence

    Diankun Wu, Fangfu Liu, Yi-Hsin Hung, and Yueqi Duan. Spatial-mllm: Boosting mllm capabilities in visual-based spatial intelligence. arXiv preprint arXiv:2505.23747, 2025 a

    Pith/arXiv arXiv 2025

  42. [42]

    Afforddp: Generalizable diffusion policy with transferable affordance

    Shijie Wu, Yihang Zhu, Yunao Huang, Kaizhen Zhu, Jiayuan Gu, Jingyi Yu, Ye Shi, and Jingya Wang. Afforddp: Generalizable diffusion policy with transferable affordance. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 6971--6980, 2025 b

    2025

  43. [43]

    Magma: A foundation model for multimodal ai agents

    Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, et al. Magma: A foundation model for multimodal ai agents. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 14203--14214, 2025

    2025

  44. [44]

    Roboengine: Plug-and-play robot data augmentation with semantic robot segmentation and background generation

    Chengbo Yuan, Suraj Joshi, Shaoting Zhu, Hang Su, Hang Zhao, and Yang Gao. Roboengine: Plug-and-play robot data augmentation with semantic robot segmentation and background generation. arXiv preprint arXiv:2503.18738, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    Robopoint: A vision-language model for spatial affordance prediction for robotics

    Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406.10721, 2024

    Pith/arXiv arXiv 2024

  46. [46]

    3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations

    Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. arXiv preprint arXiv: 2403.03954, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Vsr: a unified framework for document layout analysis combining vision, semantics and relations

    Peng Zhang, Can Li, Liang Qiao, Zhanzhan Cheng, Shiliang Pu, Yi Niu, and Fei Wu. Vsr: a unified framework for document layout analysis combining vision, semantics and relations. In International conference on document analysis and recognition, pp.\ 115--130. Springer, 2021

    2021

  48. [48]

    Learning fine-grained bimanual manipulation with low-cost hardware

    Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023

    Pith/arXiv arXiv 2023

  49. [49]

    Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness

    Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness. arXiv preprint arXiv:2409.18125, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    Learning generalizable manipulation policies with object-centric 3d representations

    Yifeng Zhu, Zhenyu Jiang, Peter Stone, and Yuke Zhu. Learning generalizable manipulation policies with object-centric 3d representations. arXiv preprint arXiv:2310.14386, 2023

    Pith/arXiv arXiv 2023

  51. [51]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pp.\ 2165--2183. PMLR, 2023

    2023

  52. [52]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  53. [53]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  54. [54]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  55. [55]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...