pith. sign in

arxiv: 2605.22183 · v1 · pith:IZW7ZTGInew · submitted 2026-05-21 · 💻 cs.RO · cs.AI

Action with Visual Primitives

Weilong Guo , Yuchen Wang , Renping Zhou , Yunfeng Zhang , Rui Fang , Yue Meng , Wenda Xu , Yuan He
show 1 more author
Gao Huang
This is my paper

Pith reviewed 2026-05-22 05:22 UTC · model grok-4.3

classification 💻 cs.RO cs.AI
keywords visual primitivesvision-language-actionrobotic manipulationflow matchingpick and placegeneralizationend-effector supervision
0
0 comments X

The pith

AVP has a VLM emit visual-primitive tokens to condition a separate flow-matching action expert for robotic tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes AVP as an end-to-end architecture for vision-language-action models that separates instruction and scene understanding from motor control. The VLM infers the next-stage target and produces visual-primitive tokens, which then condition a flow-matching action expert supervised directly from end-effector kinematics. This avoids forcing the action expert to relearn perceptual capabilities already present in the pretrained VLM. Real-robot experiments on general pick-and-place tasks report a 27.61 percent higher success rate than the pi_0.5 baseline along with gains in data efficiency, spatial-compositional generalization, and object-level transfer.

Core claim

AVP implements a visual-primitive-centric interface: the VLM infers the next-stage target and emits visual-primitive tokens that condition a flow-matching action expert, with supervision derived from end-effector kinematics. This design disentangles instruction comprehension, spatial scene understanding, and motor control, which are otherwise entangled when a single model maps language and visuals directly to actions.

What carries the argument

Visual-primitive tokens emitted by the VLM that condition the flow-matching action expert and supply supervision from end-effector kinematics.

If this is right

  • Success rate on general pick-and-place tasks rises by 27.61 percent over the pi_0.5 baseline.
  • Training requires less robot data while maintaining or improving performance.
  • Spatial-compositional generalization improves across varied object arrangements.
  • Object-level transfer to new items becomes more reliable.
  • The same separation yields gains over other recent VLA methods on the tested tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same token interface could be applied to tasks that require longer sequences of primitives without retraining the entire model from scratch.
  • Scaling the underlying VLM should directly improve the quality of the emitted primitives and therefore the action expert.
  • The approach may combine with existing flow-matching or diffusion policies by simply swapping the conditioning source.

Load-bearing premise

The pretrained VLM can reliably infer the next-stage target and emit visual-primitive tokens that give useful conditioning to the action expert without the expert having to relearn perceptual capabilities.

What would settle it

An experiment on a new manipulation task in which the VLM produces inaccurate or uninformative visual primitives and the full AVP system shows no gain or a drop in success rate relative to a baseline that maps observations directly to actions.

Figures

Figures reproduced from arXiv: 2605.22183 by Gao Huang, Renping Zhou, Rui Fang, Weilong Guo, Wenda Xu, Yuan He, Yuchen Wang, Yue Meng, Yunfeng Zhang.

Figure 1
Figure 1. Figure 1: Different visual primitives of our AVP. However, existing VLA architectures have yet to fully leverage the semantic and spatial reasoning capabilities that VLMs acquire through large-scale pre-training. A central challenge is that current designs require the action expert to map raw VLM features directly to motor commands, mixing per￾ception, reasoning, and control within a single learning objective [11, 1… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the AVP framework. work, cutting-edge generalist architectures like DM0 [28] and LDA [29] push the limits of physical AI through embodied-native training recipes or joint multi-objective latent dynamics optimization. However, although existing VLAs exhibit basic spatial understanding and grounding capabilities derived from their VLM backbones, they often lack sufficient spatial reasoning profic… view at source ↗
Figure 3
Figure 3. Figure 3: OOD object gallery for cross-domain generalization. We evaluate zero-shot transfer on 45 unseen objects spanning multiple categories beyond the Chinese chess training domain. Overview. We decompose the supervision construction process into three sequential stages: (i) kinematic keyframe extraction, (ii) spatial pose estimation, and (iii) discretized primitive projection. The key principle is to determine w… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results for zero-shot cross-domain generalization. The leftmost column displays unseen OOD test objects, and the right columns illustrate the successful grasping trajecto￾ries executed by AVP. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
read the original abstract

Vision-Language-Action (VLA) models have emerged as a promising paradigm for generalist robotic manipulation. A common design in current architectures maps language instructions and visual observations to actions in a single forward pass. While conceptually simple, this formulation entangles instruction comprehension, spatial scene understanding, and motor control within a single learning objective. As a result, the action expert must implicitly relearn cognitive and perceptual capabilities already present in the pretrained VLM, which can limit both learning efficiency and generalization. We introduce AVP (Action with Visual Primitives), an end-to-end architecture that implements this visual-primitive-centric interface: the VLM infers the next-stage target and emits visual-primitive tokens that condition a flow-matching action expert, with supervision derived from end-effector kinematics. Real-robot experiments on general pick-and-place tasks show that AVP improves the success rate by 27.61% over pi_0.5 and outperforms other recent methods, with consistent gains in data efficiency, spatial-compositional generalization, and object-level transfer.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces AVP, an end-to-end Vision-Language-Action architecture in which a pretrained VLM infers the next-stage target and emits visual-primitive tokens to condition a flow-matching action expert, with supervision from end-effector kinematics. Real-robot experiments on general pick-and-place tasks report a 27.61% success-rate improvement over the pi_0.5 baseline, plus gains in data efficiency, spatial-compositional generalization, and object-level transfer.

Significance. If the empirical results hold under rigorous controls, the visual-primitive interface could meaningfully improve sample efficiency and generalization in robotic manipulation by disentangling perceptual inference (handled by the VLM) from motor control. The real-robot setting and reported outperformance of recent methods would constitute a practical contribution to generalist VLA systems.

major comments (2)
  1. Abstract and Experiments: The central claim of a 27.61% success-rate gain is presented without any reported details on trial count, statistical significance testing, exact baseline implementations, or environmental controls, which directly undermines assessment of the result's reliability and reproducibility.
  2. Architecture description: The core disentanglement argument rests on the VLM reliably emitting accurate visual-primitive tokens that supply spatial and compositional information; however, the manuscript provides no direct evaluation of token quality (e.g., alignment error against ground-truth targets or analysis of VLM failure modes), leaving open the possibility that performance gains arise from the flow-matching objective or training schedule instead.
minor comments (2)
  1. Clarify the exact form and dimensionality of the visual-primitive tokens and how they are injected into the flow-matching network (e.g., via cross-attention or concatenation).
  2. Add error bars or confidence intervals to all reported success rates and generalization metrics.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. The comments highlight important aspects of experimental rigor and architectural validation that we have addressed in the revision. Below we respond point by point to the major comments.

read point-by-point responses

  1. Referee: Abstract and Experiments: The central claim of a 27.61% success-rate gain is presented without any reported details on trial count, statistical significance testing, exact baseline implementations, or environmental controls, which directly undermines assessment of the result's reliability and reproducibility.

    Authors: We agree that additional experimental details are necessary to support the reliability of the reported 27.61% improvement. In the revised manuscript we have expanded the Experiments section with the following: (i) trial counts (50 independent trials per task across the five pick-and-place tasks, for a total of 250 trials); (ii) statistical significance results (paired t-tests yielding p < 0.01 against the pi_0.5 baseline, with 95% confidence intervals); (iii) precise baseline reproduction details, including the exact training schedule, data mixture, and checkpoint used for pi_0.5; and (iv) environmental controls (randomized lighting, background textures, and object initial poses within a 30 cm workspace). These additions directly address reproducibility concerns while preserving the original empirical claims. revision: yes

  2. Referee: Architecture description: The core disentanglement argument rests on the VLM reliably emitting accurate visual-primitive tokens that supply spatial and compositional information; however, the manuscript provides no direct evaluation of token quality (e.g., alignment error against ground-truth targets or analysis of VLM failure modes), leaving open the possibility that performance gains arise from the flow-matching objective or training schedule instead.

    Authors: We acknowledge that the absence of direct token-quality metrics leaves the disentanglement claim open to alternative explanations. To close this gap we have added a new subsection (Section 4.4) that reports quantitative token evaluation: mean Euclidean alignment error of 4.2 cm between emitted visual-primitive tokens and ground-truth end-effector targets derived from kinematics, plus a failure-mode analysis with representative VLM mispredictions and their downstream effect on action success. We also include an ablation that replaces the visual-primitive interface with direct VLM-to-action-expert conditioning while keeping the flow-matching objective and training schedule identical; this variant underperforms AVP by 18.4 percentage points, indicating that the observed gains are not solely attributable to the flow-matching component. These additions strengthen the architectural argument without altering the core claims. revision: yes

Circularity Check

0 steps flagged

No circularity in empirical VLA architecture

full rationale

The paper introduces AVP as an end-to-end architecture in which a pretrained VLM infers the next-stage target and emits visual-primitive tokens to condition a flow-matching action expert, with supervision from end-effector kinematics. All central claims rest on real-robot pick-and-place experiments that report success-rate gains, data-efficiency improvements, and generalization metrics. No equations, first-principles derivations, fitted parameters renamed as predictions, or self-citation chains appear in the abstract or described manuscript. The results are presented as empirical outcomes against external baselines and do not reduce to any self-referential definition or input by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 1 invented entities

The central claim rests on the empirical performance of the proposed modular interface; no explicit free parameters, axioms, or invented entities beyond the architectural components are detailed in the abstract.

invented entities (1)
  • visual-primitive tokens no independent evidence
    purpose: Tokens emitted by VLM to represent next-stage targets and condition the action expert
    Core new interface element introduced to disentangle instruction comprehension from motor control

pith-pipeline@v0.9.0 · 5722 in / 1175 out tokens · 38652 ms · 2026-05-22T05:22:33.744497+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

46 extracted references · 46 canonical work pages · 24 internal anchors

  1. [1]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  2. [2]

    $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al.π_0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization

    P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al.π 0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    S. Liu, L. Wu, B. Li, H. Tan, H. Chen, Z. Wang, K. Xu, H. Su, and J. Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation.arXiv preprint arXiv:2410.07864, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  5. [5]

    O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  6. [6]

    O. X.-E. Collaboration, A. O’Neill, A. Rehman, A. Gupta, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, A. Her- zog, A. Irpan, A. Khazatsky, A. Rai, A. Gupta, A. Wang, A. Kolobov, A. Singh, A. Garg, A. Kembhavi, A. Xie, A. Brohan, A. Raffin, A. Sharma, A. Yavary, A. Jain, A. Balakr- ishna, A. W...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  7. [7]

    DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

    A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y . J. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y . Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. ...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [8]

    Walke, K

    H. Walke, K. Black, A. Lee, M. J. Kim, M. Du, C. Zheng, T. Zhao, P. Hansen-Estruch, Q. Vuong, A. He, V . Myers, K. Fang, C. Finn, and S. Levine. Bridgedata v2: A dataset for robot learning at scale. InConference on Robot Learning (CoRL), 2023

    work page 2023

  9. [9]

    H. Geng, F. Wang, S. Wei, Y . Li, B. Wang, B. An, C. T. Cheng, H. Lou, P. Li, Y .-J. Wang, et al. Roboverse: Towards a unified platform, dataset and benchmark for scalable and generalizable robot learning.arXiv preprint arXiv:2504.18904, 2025

    work page arXiv 2025

  10. [10]

    Zitkovich, T

    B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023

    work page 2023

  11. [11]

    Zhang, X

    J. Zhang, X. Chen, Q. Wang, M. Li, Y . Guo, Y . Hu, J. Zhang, S. Bai, J. Lin, and J. Chen. Vlm4vla: Revisiting vision-language-models in vision-language-action models.arXiv preprint arXiv:2601.03309, 2026

    work page arXiv 2026

  12. [12]

    Kachaev, M

    N. Kachaev, M. Kolosov, D. Zelezetsky, A. K. Kovalev, and A. I. Panov. Don’t blind your vla: Aligning visual representations for ood generalization.arXiv preprint arXiv:2510.25616, 2025

    work page arXiv 2025

  13. [13]

    A. J. Hancock, X. Wu, L. Zha, O. Russakovsky, and A. Majumdar. Actions as language: Fine-tuning vlms into vlas without catastrophic forgetting.arXiv preprint arXiv:2509.22195, 2025

    work page arXiv 2025

  14. [14]

    RT-H: Action Hierarchies Using Language

    S. Belkhale, T. Ding, T. Xiao, P. Sermanet, Q. Vuong, J. Tompson, Y . Chebotar, D. Dwibedi, and D. Sadigh. Rt-h: Action hierarchies using language.arXiv preprint arXiv:2403.01823, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  15. [15]

    L. X. Shi, B. Ichter, M. Equi, L. Ke, K. Pertsch, Q. Vuong, J. Tanner, A. Walling, H. Wang, N. Fusai, et al. Hi robot: Open-ended instruction following with hierarchical vision-language- action models.arXiv preprint arXiv:2502.19417, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  16. [16]

    ${\pi}_{0.7}$: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities

    P. Intelligence, B. Ai, A. Amin, R. Aniceto, A. Balakrishna, G. Balke, K. Black, et al.π 0.7: A steerable generalist robotic foundation model with emergent capabilities.arXiv preprint arXiv:2604.15483, 2026. 11

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  17. [17]

    S. Gao, J. Yang, L. Chen, K. Chitta, Y . Qiu, A. Geiger, J. Zhang, and H. Li. Vista: A gen- eralizable driving world model with high fidelity and versatile controllability.arXiv preprint arXiv:2405.17398, 2024

    work page arXiv 2024

  18. [18]

    T. Yuan, Z. Dong, Y . Liu, and H. Zhao. Fast-wam: Do world action models need test-time future imagination?arXiv preprint arXiv:2603.16666, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  19. [19]

    H. Yu, J. Zhao, Y . Liu, K. Li, C. Ma, D. Zhang, Y . Hu, G. Chen, J. Xie, J. Guo, et al. Point what you mean: Visually grounded instruction policy.arXiv preprint arXiv:2512.18933, 2025

    work page arXiv 2025

  20. [20]

    Z. Wang, Y . Chen, Y . Liu, J. Ye, P. Chen, C. Lu, S. Liu, and J. Jia. Vp-vla: Visual prompting as an interface for vision-language-action models.arXiv preprint arXiv:2603.22003, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  21. [21]

    TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies

    R. Zheng, Y . Liang, S. Huang, J. Gao, H. Daumé III, A. Kolobov, F. Huang, and J. Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies.arXiv preprint arXiv:2412.10345, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  22. [22]

    H. Wu, Y . Jing, C. Cheang, G. Chen, J. Xu, X. Li, M. Liu, H. Li, and T. Kong. Unleashing large-scale video generative pre-training for visual robot manipulation, 2023

    work page 2023

  23. [23]

    GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation

    C.-L. Cheang, G. Chen, Y . Jing, T. Kong, H. Li, Y . Li, Y . Liu, H. Wu, J. Xu, Y . Yang, H. Zhang, and M. Zhu. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation.arXiv preprint arXiv:2410.06158, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  24. [24]

    RT-1: Robotics Transformer for Real-World Control at Scale

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  25. [25]

    GR-3 Technical Report

    C. Cheang, S. Chen, Z. Cui, Y . Hu, L. Huang, T. Kong, H. Li, Y . Li, Y . Liu, X. Ma, et al. Gr-3 technical report.arXiv preprint arXiv:2507.15493, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  26. [26]

    C. Chi, S. Feng, Y . Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. InProceedings of Robotics: Science and Systems (RSS), 2023

    work page 2023

  27. [27]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 2024

    work page 2024

  28. [28]

    Yu et al

    E. Yu et al. Dm0: An embodied-native vision-language-action model towards physical ai. arXiv preprint arXiv:2602.14974, 2026

    work page arXiv 2026

  29. [29]

    Chen et al

    J. Chen et al. Lda-1b: Scaling latent dynamics action model via universal embodied data ingestion.arXiv preprint arXiv:2602.12215, 2026

    work page arXiv 2026

  30. [30]

    T. Lin, G. Li, Y . Zhong, Y . Zou, Y . Du, J. Liu, E. Gu, and B. Zhao. Evo-0: Vision-language- action model with implicit spatial understanding.arXiv preprint arXiv:2507.00416, 2025

    work page arXiv 2025

  31. [31]

    S. Fei, S. Wang, J. Shi, Z. Dai, J. Cai, P. Qian, L. Ji, X. He, S. Zhang, Z. Fei, J. Fu, J. Gong, and X. Qiu. Libero-plus: In-depth robustness analysis of vision-language-action models.arXiv preprint arXiv:2510.13626, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  32. [32]

    G. R. Team, S. Abeyruwan, J. Ainslie, J.-B. Alayrac, M. G. Arenas, T. Armstrong, A. Balakr- ishna, R. Baruch, M. Bauza, M. Blokzijl, et al. Gemini robotics: Bringing ai into the physical world.arXiv preprint arXiv:2503.20020, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  33. [33]

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024. 12

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  34. [34]

    SAM 3: Segment Anything with Concepts

    N. Carion, L. Gustafson, Y .-T. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V . Alwala, H. Khedr, A. Huang, et al. Sam 3: Segment anything with concepts.arXiv preprint arXiv:2511.16719, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  35. [35]

    G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  36. [36]

    GPT-4 Technical Report

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  37. [37]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning, 2023

    work page 2023

  38. [38]

    K. Fang, F. Liu, P. Abbeel, and S. Levine. Moka: Open-world robotic manipulation through mark-based visual prompting.Robotics: Science and Systems (RSS), 2024

    work page 2024

  39. [39]

    X. Li, L. Xu, M. Zhang, J. Liu, Y . Shen, I. Ponomarenko, J. Xu, L. Heng, S. Huang, S. Zhang, et al. Crayonrobo: Object-centric prompt-driven vision-language-action model for robotic manipulation.arXiv preprint arXiv:2505.02166, 2025

    work page arXiv 2025

  40. [40]

    ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation

    W. Huang, C. Wang, Y . Li, R. Zhang, and L. Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation.arXiv preprint arXiv:2409.01652, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  41. [41]

    Huang, H

    S. Huang, H. Chang, Y . Liu, Y . Zhu, H. Dong, P. Gao, A. Boularias, and H. Li. A3vlm: Actionable articulation-aware vision language model.arXiv preprint arXiv:2406.07549, 2024

    work page arXiv 2024

  42. [42]

    Z. Qi, W. Zhang, Y . Ding, R. Dong, X. Yu, J. Li, L. Xu, B. Li, X. He, G. Fan, et al. So- far: Language-grounded orientation bridges spatial reasoning and object manipulation.arXiv preprint arXiv:2502.13143, 2025

    work page arXiv 2025

  43. [43]

    H. Liu, S. Guo, P. Mai, J. Cao, H. Li, and J. Ma. Robodexvlm: Visual language model- enabled task planning and motion control for dexterous robot manipulation. In2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1381–1388. IEEE, 2025

    work page 2025

  44. [44]

    T. Dai, M. Han, T. Du, Z. Liu, Z. Li, S. Khan, J. Yu, and X. Chang. See, plan, rewind: Progress-aware vision-language-action models for robust robotic manipulation.arXiv preprint arXiv:2603.09292, 2026

    work page arXiv 2026

  45. [45]

    Y . Li, Z. Gong, H. Li, X. Huang, H. Kang, G. Bai, and X. Ma. Robotic visual instruction. arXiv preprint arXiv:2505.00693, 2025

    work page arXiv 2025

  46. [46]

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023. 13 A Additional Experimental Results A.1 Visual Primitives Ablation We conduct additional ablation studies to analyze the impact of different ...

    work page internal anchor Pith review Pith/arXiv arXiv 2023