Pith. sign in

REVIEW 5 major objections 4 minor 38 references

An RGB-only robot policy can internalize target-object 3D priors during training and use them at test time, improving long-horizon and occluded manipulation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 01:06 UTC pith:Y2EWKI5G

load-bearing objection Useful object-centric alignment recipe for VLAs, but the 3D-specific mechanism is not actually demonstrated; needs a 2D-teacher ablation and a clear description of SAM3D's single-image features. the 5 major comments →

arxiv 2607.25912 v1 pith:Y2EWKI5G submitted 2026-07-28 cs.RO cs.AI

SAM3D-Guided Object-Centric Representation Alignment for Vision-Language-Action Models

Zonghe Liu (1) , Shanyuan Jie (2) , Xiaoquan Sun (3) , Chen Cao (1) , Zetian Xu (1) , Zongsheng Liu (4) , Jiayu Chen (1 , 5) ((1) University of Hong Kong
show 7 more authors
(2) Shenzhen Institutes of Advanced Technology Chinese Academy of Sciences (3) Huazhong University of Science Technology (4) Beijing University of Aeronautics Astronautics (5) Infiforce)
This is my paper
classification cs.RO cs.AI
keywords Vision-Language-Action modelsobject-centric 3D priorsrepresentation alignmentSAM3Drobot manipulationlong-horizon tasksocclusion robustnesspi0
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The authors ask whether a vision-language-action model that only receives RGB images and language instructions at test time can still benefit from 3D object understanding. Their answer is yes: during training, they use a frozen 3D segmentation model to extract object-centric geometric features of the task-relevant object, and align the policy's intermediate visual features to those features with a masked loss. At test time, all 3D machinery is removed, and the policy runs the original RGB-language-to-action pipeline. Across LIBERO, CALVIN, and real-world tabletop tasks, the aligned policy outperforms the base model and several 3D-enhanced baselines, with the largest gains in long-horizon and occlusion settings. If correct, this shows that explicit 3D sensors are not necessary for object-level 3D reasoning in manipulation, as long as the right priors are distilled during training.

Core claim

The central discovery is that object-centric 3D information can be transferred from a frozen teacher into a VLA policy through representation alignment at an intermediate layer, rather than by changing the input modality or adding 3D modules at deployment. The authors build on a base VLA and add a training-only alignment loss: subtask instructions are decomposed, target objects are detected and segmented into masks, and a frozen SAM3D teacher converts each masked image into dense 3D feature tokens. These tokens are resized to match the policy's visual-token grid, and the policy's intermediate visual features, projected into teacher space, are trained to match them only on masked object token

What carries the argument

The load-bearing machinery is a masked normalized feature-alignment loss applied to intermediate visual tokens of the VLA backbone. A frozen SAM3D teacher (a segment-anything-in-3D model that outputs dense object-level geometric features from a single image plus mask) provides the target representation; the student's intermediate features are projected into teacher space, L2-normalized, and matched only on tokens inside the task-relevant object mask. For long-horizon instructions, an LLM decomposes the task into subtasks, and each subtask supplies its own object mask, so the alignment is stage-specific. This trains the policy to encode shape and layout priors of the current target object wit

Load-bearing premise

The automatically generated subtask decompositions and object masks must be accurate enough that the SAM3D teacher features describe the correct task-relevant object; if the decomposition or segmentation is wrong, the alignment loss teaches the policy the wrong 3D prior.

What would settle it

Train the same method but replace the automatic masks with random masks or masks of a fixed non-target object; if success rates do not drop on LIBERO-Long and CALVIN, the object-centric alignment is not the source of the reported gains. Alternatively, run the real-world occlusion experiment with transparent or severely occluded objects where SAM3D is known to fail, and check whether the occlusion-setting gains persist.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Inference remains unchanged: no depth, point clouds, masks, or 3D modules are needed at test time.
  • Training-time alignment yields consistent gains on LIBERO (99.1% average success), CALVIN (4.11 average length), and real-world tabletop tasks.
  • Gains are largest in long-horizon tasks where the target object changes across subtasks, and in occlusion settings.
  • A frozen-representation probe shows that SAM3D-style object-centric 3D priors become more accurately recoverable from the aligned VLA features, supporting the proposed mechanism.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If this generalizes, object-centric 3D distillation could reduce or eliminate the need for depth sensors and RGB-D pipelines in many manipulation settings, since the policy acquires object-level geometry during training rather than at deployment.
  • A direct test of the mechanism's sensitivity would be to vary the quality of the automatically generated masks and measure how LIBERO success degrades; this would quantify how much of the gain depends on segmentation accuracy.
  • The subtask-decomposition step suggests a broader principle: stage-specific auxiliary supervision targets, rather than global scene-level alignment alone, may be what matters for long-horizon policies, and this could transfer to other auxiliary signals beyond SAM3D.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes SAM3D-VLA, a training-time framework that aligns intermediate visual features of the π0 vision-language-action model with object-centric 3D features extracted by a frozen SAM3D teacher. Task-relevant objects are grounded and segmented automatically, and for long-horizon tasks instructions are decomposed into subtasks with per-stage object masks. The alignment loss is a masked normalized MSE between projected student features and resampled teacher features. At inference only RGB images and language are used. Experiments on LIBERO, CALVIN, and a real Piper-X robot report consistent improvements over strong baselines, e.g., 99.1% average LIBERO success and CALVIN average length 4.11. The paper also includes a probing experiment intended to show that SAM3D-style features become more recoverable from the aligned VLA features.

Significance. If validated, the contribution is practically valuable: it demonstrates a way to inject object-centric 3D priors into an RGB-only VLA without changing the inference pipeline or requiring depth/point clouds at test time. The evaluation covers standard benchmarks and a real robot, and the idea of using a frozen teacher to supervise intermediate representations is timely. The manuscript is clearly written and the main idea is appealing. However, the central claim that the gains come specifically from 3D priors is not yet supported by the experiments: there is no ablation replacing SAM3D with a 2D teacher, and the described use of SAM3D on single images is inconsistent with the cited method. The lack of error bars, seeds, and per-task trial counts further weakens the quantitative evidence. The probing experiment is a reasonable sanity check but is partly self-referential and does not establish causal relevance.

major comments (5)
  1. [§3.1, Eq. (3)-(4), Table 1] The central claim is that object-centric 3D priors drive the gains, but no ablation replaces SAM3D with a 2D object-centric teacher (e.g., DINOv2 or SAM features). Since Lalign includes mask-weighted alignment and the pipeline also includes subtask decomposition (§3.2), the observed improvements could come from the object-centric attention or the subtask-aware supervision rather than from 3D shape/layout information. Please add a 2D-teacher ablation to isolate the 3D contribution.
  2. [§3.1, SAM3D usage] The paper states SAM3D operates on single images and flattens multi-view observations, but the cited SAM3D [17] is a 3D scene segmentation method that requires a reconstructed scene. The exact per-image feature extraction mode is unspecified. If the teacher is not actually producing 3D features in this single-image setting, the claim of injecting '3D priors' is unsupported. Please clarify the SAM3D feature extraction, or correct the description.
  3. [Tables 1-3] All reported results are single numbers without variance, seeds, or per-task trial counts. This is especially problematic for Table 3, where the difference between 90% and 92% could be a few trials. The realism of the real-world comparisons is also unclear because no protocol details (number of rollouts, randomization, failure modes) are given. Please report standard deviations over multiple seeds and per-task trial counts.
  4. [Eq. (4) and layer m] Two key hyperparameters—alignment weight α and the student layer m from which features are extracted—are never ablated or even given values. Since the method's effectiveness could hinge on these choices, a sensitivity study is needed to show the results are robust and to enable reproduction.
  5. [§3.1, Eq. (5)] The probing experiment trains a probe on the same intermediate features that were aligned to SAM3D targets and then tests recovery of those same SAM3D targets. This is partly self-referential and does not demonstrate that the recovered features are causally responsible for action success. The paper should either add a negative control (e.g., probing for random features) or explicitly frame this experiment as a sanity check, not independent evidence for the mechanism.
minor comments (4)
  1. [Throughout] Typography and grammar errors: 'As showen' (Fig. 4 caption), 'Infiforce' (affiliation), 'A verage' (Table 3), inconsistent 'Piper-x' vs 'Piper-X'. A careful proofread is needed.
  2. [Fig. 1 and Fig. 2] The notation for views and masks in the figure is dense and could be clarified; in particular, the mapping from masked object to SAM3D tokens is not obvious from the figure alone.
  3. [§4, CALVIN] The paper says 500 rollouts are used for CALVIN evaluation but does not report whether the same checkpoints/seeds were used for all baselines; please specify the evaluation protocol.
  4. [References] Some references appear to be future-dated (e.g., [4], [5], [14]) or are listed with inconsistent formats; please ensure all bibliographic details are correct and consistent.

Circularity Check

1 steps flagged

The probing experiment reduces to the alignment loss by construction, but the central benchmark claims rest on independent success metrics.

specific steps
  1. fitted input called prediction [Section 3.1, Eq. (5), 'Frozen-representation Probing Adapter']
    "After policy training, we freeze the entire VLA backbone and train only a lightweight two-layer MLP probe Pω on top of the same intermediate visual features St used for alignment. The probe predicts the spatially resampled SAM3D target ¯Tt: ... Lprobe = MSE(Norm(˜Tt)[Mt], Norm(¯Tt)[Mt])."

    The probe target ¯Tt is the same SAM3D feature that Lalign (Eq. 3) trains the VLA features St to match: Lalign = MSE(Norm(Pφ(St))[Mt], Norm(¯Tt)[Mt]). Probing St for ¯Tt after this training therefore measures, by construction, whether the alignment objective was optimized; it cannot independently establish that 'SAM3D-style object-centric 3D priors' were internalized, and it says nothing about whether those priors cause the reported action success. It is a sanity check of the training loss, presented as a mechanism verification.

full rationale

The central empirical claims (LIBERO 99.1%, CALVIN 4.11, and real-world success rates) are task-success metrics on external benchmarks; they are not derived from the alignment loss and involve no fitted constant disguised as a prediction. The method's Lalign (Eq. 3) directly optimizes the intermediate visual features to be decodable into the SAM3D teacher features, so the probing experiment (Eq. 5) is a self-referential sanity check: the probe target and probe input are exactly the target and feature that Lalign was trained to couple. This makes the 'mechanism verification' contribution partly circular, but it is not the load-bearing evidence for the main manipulation improvements. The missing 2D-teacher ablation, which would be needed to attribute gains specifically to 3D priors, and the questionable claim that 'SAM3D operates on single images' are validity and attribution concerns, not circularity. Overall, the paper shows genuine independent benchmark gains, so the circularity score is low.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical or architectural entity; it uses existing models (SAM3D, SAM2, Grounding DINO, GPT-4o, π0) as components. Its load-bearing assumptions are domain assumptions about the usefulness and correctness of the masked 3D supervision and the alignment objective.

free parameters (2)
  • Alignment weight α = not reported
    Eq. 4 combines the flow-matching action loss with Lalign; the value of α is never given and no ablation is shown, yet the method's gains depend on it.
  • Student layer m = not reported
    Intermediate Gemma layer selected for alignment and probing; the choice affects which features are supervised and is not specified or ablated.
axioms (5)
  • domain assumption SAM3D features from a 2D image plus object mask encode object-level 3D geometric priors useful for manipulation.
    Section 3.1 treats T_t = f_SAM3D(I_t, M_t) as a frozen teacher; if these features do not carry useful 3D shape/layout information, alignment targets are meaningless.
  • domain assumption Masked normalized MSE alignment of intermediate VLA features to a frozen 3D teacher improves downstream action prediction without harming the flow-matching objective.
    Eqs. 3-4 assume auxiliary alignment transfers to action success; this is supported only empirically, not by derivation or ablation isolating the alignment term.
  • domain assumption Automatic object grounding and segmentation produce correct task-relevant masks for each subtask.
    Section 3.2 and Fig. 4 rely on Grounding DINO/YOLO + SAM2; the Limitations section (Sec. 6) acknowledges that mask errors inject noisy supervision.
  • domain assumption GPT-4o subtask decomposition matches the human-annotated task structure for long-horizon instructions.
    Subtask-aware processing uses LLM decomposition to assign per-stage object masks; no accuracy analysis of the decomposition is provided.
  • domain assumption Bilinear interpolation of the SAM3D token grid preserves spatial correspondence with π0 visual tokens across camera views.
    Section 3.1 resizes teacher features to student token resolution; misalignment would spread object features to wrong tokens and weaken the mask-specific supervision.

pith-pipeline@v1.3.0-alltime-deepseek · 10232 in / 9082 out tokens · 82801 ms · 2026-08-01T01:06:52.851421+00:00 · methodology

0 comments
read the original abstract

Vision-Language-Action (VLA) models have shown strong potential for general robot manipulation, but most existing models rely on 2D visual-language backbones and lack fine-grained 3D understanding of target objects, especially under occlusion, pose variation, scale changes, and precise spatial interaction. We propose an object-centric 3D representation alignment framework built upon $\pi_0$, using SAM3D as a frozen 3D teacher to provide target-object 3D priors during training. Specifically, we localize task-relevant objects with object recognition models, generate corresponding object masks, and use SAM3D to extract dense object-level 3D representations, which are aligned with intermediate visual features of $\pi_0$. This enables the policy to internalize target-object 3D information while preserving the original RGB-language-to-action inference pipeline without requiring depth, point clouds, masks, SAM3D, or additional 3D modules at test time. Simulation experiments show consistent improvements, achieving 99.1\% on LIBERO and an average length of 4.11 on CALVIN. Real-world experiments further demonstrate that our method is particularly effective in long-horizon manipulation scenarios where the robot must focus on different target objects across multiple subtasks.

Figures

Figures reproduced from arXiv: 2607.25912 by (2) Shenzhen Institutes of Advanced Technology, (3) Huazhong University of Science, (4) Beijing University of Aeronautics, 5) ((1) University of Hong Kong, (5) Infiforce), Astronautics, Chen Cao (1), Chinese Academy of Sciences, Jiayu Chen (1, Shanyuan Jie (2), Technology, Xiaoquan Sun (3), Zetian Xu (1), Zonghe Liu (1), Zongsheng Liu (4).

Figure 1
Figure 1. Figure 1: Overview of SAM3D-VLA. We propose an object-centric 3D alignment framework that uses SAM3D as a frozen teacher during training. High-level instructions are decomposed into sub￾tasks, and task-relevant object masks are used for SAM3D feature extraction. The extracted 3D features are spatially resampled and dimensionally projected to align with intermediate VLA repre￾sentations, while inference follows the o… view at source ↗
Figure 2
Figure 2. Figure 2: The framework of SAM3D-VLA. Task-relevant object masks are generated from subtask instructions and fed to a frozen SAM3D teacher to provide object-centric 3D supervision for inter￾mediate VLA features. We further use a frozen-representation probing adapter to verify whether SAM3D priors are encoded in the learned VLA representations. In our framework, we preserve the original π0 formulation, where the poli… view at source ↗
Figure 3
Figure 3. Figure 3: Real-world experimental setup and Mask processing. (a) Piper-x Platform. (b) Workspace. Dataset Processing: The target object is grounded and masked by open-source mod￾els. SAM3D-guided object-centric alignment helps the policy encode target-object 3D priors for se￾quential manipulation, while preserving the original π0 RGB-language-to-action inference pipeline without additional test-time inputs or module… view at source ↗
Figure 4
Figure 4. Figure 4: Subtask-aware object-centric data processing. High-level long-horizon instructions are decomposed into subtasks, and each subtask is associated with its corresponding target object mask. During training, the model receives stage-specific SAM3D supervision, encouraging it to focus on different task-relevant objects across manipulation steps. 5 Conclusion In this paper, we propose SAM3D-VLA, an object-centri… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 13 linked inside Pith

  1. [1]

    Zitkovich, T

    B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023

    2023

  2. [2]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al.π 0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    C. Li, J. Wen, Y . Peng, Y . Peng, and Y . Zhu. Pointvla: Injecting the 3d world into vision- language-action models.IEEE Robotics and Automation Letters, 11(3):2506–2513, 2026

    2026

  5. [5]

    P. Li, Y . Chen, H. Wu, X. Ma, X. Wu, Y . Huang, L. Wang, T. Kong, and T. Tan. Bridgevla: Input-output alignment for efficient 3d manipulation learning with vision-language models. Advances in Neural Information Processing Systems, 38:63635–63673, 2026

    2026

  6. [6]

    L. Sun, B. Xie, Y . Liu, H. Shi, T. Wang, and J. Cao. Geovla: Empowering 3d representations in vision-language-action models.arXiv preprint arXiv:2508.09071, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Bhat, Y .-H

    V . Bhat, Y .-H. Lan, P. Krishnamurthy, R. Karri, and F. Khorrami. 3d cavla: Leveraging depth and 3d context to generalize vision language action models for unseen tasks.arXiv preprint arXiv:2505.05800, 2025

    arXiv 2025

  8. [8]

    D. Qu, H. Song, Q. Chen, Y . Yao, X. Ye, Y . Ding, Z. Wang, J. Gu, B. Zhao, D. Wang, et al. Spatialvla: Exploring spatial representations for visual-language-action model.arXiv preprint arXiv:2501.15830, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    F. Li, W. Song, H. Zhao, J. Wang, P. Ding, D. Wang, L. Zeng, and H. Li. Spatial forcing: Implicit spatial representation alignment for vision-language-action model.arXiv preprint arXiv:2510.12276, 2025

    arXiv 2025

  10. [10]

    W. Song, Z. Zhou, H. Zhao, J. Chen, P. Ding, H. Yan, Y . Huang, F. Tang, D. Wang, and H. Li. Reconvla: Reconstructive vision-language-action model as effective robot perceiver. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 18549– 18557, 2026

    2026

  11. [11]

    Huang, X

    H. Huang, X. Chen, Y . Chen, H. Li, X. Han, Z. Wang, T. Wang, J. Pang, and Z. Zhao. Roboground: Robotic manipulation with grounded vision-language priors. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 22540–22550, 2025

    2025

  12. [12]

    Zawalski, W

    M. Zawalski, W. Chen, K. Pertsch, O. Mees, C. Finn, and S. Levine. Robotic control via embodied chain-of-thought reasoning.arXiv preprint arXiv:2407.08693, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    S. Deng, M. Yan, S. Wei, H. Ma, Y . Yang, J. Chen, Z. Zhang, T. Yang, X. Zhang, W. Zhang, et al. Graspvla: a grasping foundation model pre-trained on billion-scale synthetic action data. arXiv preprint arXiv:2505.03233, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Y . Tian, Q. Ye, and D. Doermann. Yolov12: Attention-centric real-time object detectors. Advances in neural information processing systems, 38:78433–78457, 2026

    2026

  15. [15]

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pages 38–55. Springer, 2024. 9

    2024

  16. [16]

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos. InInternational Confer- ence on Learning Representations, volume 2025, pages 28085–28128, 2025

    2025

  17. [17]

    Y . Yang, X. Wu, T. He, H. Zhao, and X. Liu. Sam3d: Segment anything in 3d scenes.arXiv preprint arXiv:2306.03908, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Z. Li, L. Ren, J. Yang, Y . Zhao, X. Wu, Z. Xu, X. Bai, and H. Zhao. Vip: Vision instructed pre-training for robotic manipulation.arXiv preprint arXiv:2410.07169, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    H. Chen, B. Sun, A. Zhang, M. Pollefeys, and S. Leutenegger. Vidbot: Learning generalizable 3d actions from in-the-wild 2d human videos for zero-shot robotic manipulation. InProceed- ings of the Computer Vision and Pattern Recognition Conference, pages 27661–27672, 2025

    2025

  20. [20]

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre- training. InProceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023

    2023

  21. [21]

    G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivi `ere, M. S. Kale, J. Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 44(10-11):1684–1704, 2025

    2025

  23. [23]

    O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Z. Hou, T. Zhang, Y . Xiong, H. Duan, H. Pu, R. Tong, C. Zhao, X. Zhu, Y . Qiao, J. Dai, et al. Dita: Scaling diffusion transformer for generalist vision-language-action policy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7686– 7697, 2025

    2025

  25. [25]

    Q. Zhao, Y . Lu, M. J. Kim, Z. Fu, Z. Zhang, Y . Wu, Z. Li, Q. Ma, S. Han, C. Finn, et al. Cot- vla: Visual chain-of-thought reasoning for vision-language-action models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 1702–1713, 2025

    2025

  26. [26]

    Q. Bu, Y . Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li. Univla: Learning to act anywhere with task-centric latent actions.arXiv preprint arXiv:2505.06111, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    M. J. Kim, C. Finn, and P. Liang. Fine-tuning vision-language-action models: Optimizing speed and success.arXiv preprint arXiv:2502.19645, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Y . Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation.Advances in neural information processing systems, 36:9156–9172, 2023

    2023

  29. [29]

    Black, M

    K. Black, M. Nakamoto, P. Atreya, H. Walke, C. Finn, A. Kumar, and S. Levine. Zero- shot robotic manipulation with pre-trained image-editing diffusion models. InInternational Conference on Learning Representations, volume 2024, pages 33431–33452, 2024

    2024

  30. [30]

    H. Wu, Y . Jing, C. Cheang, G. Chen, J. Xu, X. Li, M. Liu, H. Li, and T. Kong. Unleashing large- scale video generative pre-training for visual robot manipulation. InInternational Conference on Learning Representations, volume 2024, pages 10641–10662, 2024

    2024

  31. [31]

    Y . Wen, J. Lin, Y . Zhu, J. Han, H. Xu, S. Zhao, and X. Liang. Vidman: Exploiting implicit dynamics from video diffusion model for effective robot manipulation.Advances in Neural Information Processing Systems, 37:41051–41075, 2024. 10

    2024

  32. [32]

    Q. Bu, J. Zeng, L. Chen, Y . Yang, G. Zhou, J. Yan, P. Luo, H. Cui, Y . Ma, and H. Li. Closed- loop visuomotor control with generative expectation for robotic manipulation.Advances in Neural Information Processing Systems, 37:139002–139029, 2024

    2024

  33. [33]

    W. Zhao, P. Ding, Z. Min, Z. Gong, S. Bai, H. Zhao, and D. Wang. Vlas: Vision-language- action model with speech instructions for customized robot manipulation. InInternational conference on learning representations, volume 2025, pages 51676–51693, 2025

    2025

  34. [34]

    X. Li, M. Liu, H. Zhang, C. Yu, J. Xu, H. Wu, C. Cheang, Y . Jing, W. Zhang, H. Liu, et al. Vision-language foundation models as effective robot imitators. InInternational Conference on Learning Representations, volume 2024, pages 26703–26721, 2024

    2024

  35. [35]

    B. Liu, Y . Zhu, C. Gao, Y . Feng, Q. Liu, Y . Zhu, and P. Stone. Libero: Benchmarking knowl- edge transfer for lifelong robot learning.Advances in Neural Information Processing Systems, 36:44776–44791, 2023

    2023

  36. [36]

    Zheng, Y

    R. Zheng, Y . Liang, S. Huang, J. Gao, H. Daum ´e III, A. Kolobov, F. Huang, and J. Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. InInternational Conference on Learning Representations, volume 2025, pages 54277–54296, 2025

    2025

  37. [37]

    Pertsch, K

    K. Pertsch, K. Stachowicz, B. Ichter, D. Driess, S. Nair, Q. Vuong, O. Mees, C. Finn, and S. Levine. Fast: Efficient action tokenization for vision-language-action models.arXiv preprint arXiv:2501.09747, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks.IEEE Robotics and Automation Letters, 7(3):7327–7334, 2022. 11

    2022