Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Robotic Visual Instruction

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a hand-drawn sketch language of arrows, circles, colors, and numbers can replace natural language for commanding robots with pixel-level spatial precision.

desk verdict A useful new interaction primitive with a real evaluation gap: the execution-vs-comprehension dissociation needs per-trial evidence. read the letter →

arxiv 2505.00693 v3 pith:2IJT4P2K submitted 2025-05-01 cs.RO cs.AIcs.CV

Yanbang Li , Ziyang Gong , Haoyang Li , Xiaoqi Huang , Haolan Kang , Guangping Bai , Xianzheng Ma This is my paper · ORCID
classification cs.ROcs.AIcs.CV
keywords roboticvisualinstructionhuman-robotinteractionvision-languagemodelskeypoint-conditionedpolicyhand-drawnsketchesrobotmanipulationspatiotemporalconstraintsparameter-efficientfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes replacing natural-language robot commands with a hand-drawn visual language, RoVI, where arrows give trajectories, circles mark affordances, colors order steps, and numbers label arms. It then builds VIEW, a pipeline that turns such sketches into executable actions: a vision-language model names the task and writes Python function calls, a keypoint detector pulls pixel coordinates from the arrows and circles, and a keypoint-conditioned low-level policy moves the arm through those coordinates. To make small models practical, the authors collect a 15K-instance instruction dataset and fine-tune a 13-billion-parameter vision-language model. On eleven unseen tasks in simulation and the real world, the method reports up to 87.5% real-world execution success, outperforming language-conditioned baselines in cluttered scenes, multi-step operations, and trajectory following. The paper is establishing that a sketch, not speech, can be a precise and silent command channel for everyday robot manipulation.

What carries the argument

The load-bearing mechanism is the RoVI symbol set combined with the keypoint-conditioned policy. An arrow is formally an ordered list of pixel coordinates, starting point, intermediate waypoints, and endpoint, while a circle contributes a single affordance center; color numbers the temporal steps. The keypoint module detects these symbol parts directly in the image, so the spatial constraints come from the drawing rather than from object detection, and the pixel coordinates are lifted to 3D with depth data and converted into end-effector poses in SE(3). A cost function with a translation term and a rotation term then moves the arm from one keypoint to the next, while the vision-language model's generated Python functions decide which action each keypoint sequence represents.

What would settle it

Run the pipeline on a fixed set of RoVI sketches, then deliberately corrupt the model's natural-language task and planning responses while keeping the generated code and keypoint coordinates unchanged; if execution success drops correspondingly, the claim that execution is unaffected by perception errors is refuted, and if it stays high, keypoints and code are carrying the task.

Watch

Extended reading notes

Core claim

The paper's central claim is that a hand-drawn 2D sketch can serve as a complete robot instruction: an arrow encodes a trajectory through its tail, waypoints, and head; a circle marks where to grasp, rotate, or press; and color assigns each symbol to a step in a multi-step task. VIEW is the pipeline that reads that sketch: a vision-language model produces a task name, a step-by-step plan, and Python function calls; a keypoint module extracts the arrow and circle coordinates from the image; and a low-level policy lifts those pixel coordinates to 3D and drives the end effector through them by minimizing a translation-and-rotation cost. The reported discovery is that this combination transfers to tasks and environments the system has not seen, reaching 87.5% real-world success on trials that demand multi-step sequencing, tolerance of clutter and disturbances, and exact trajectory following. The paper further claims that execution success does not depend on the vision-language model's language comprehension, because the code path is separate from the keypoint grounding.

Load-bearing premise

The pipeline assumes the vision-language model's Python execution functions stay correct even when the same model's natural-language task and planning outputs are wrong; the paper's own numbers show a 13-billion-parameter model with weak language comprehension achieving high execution success, and that separation is the premise on which the main result rests.

Editorial extensions

If this is right

  • Users can specify exact grasp points, paths, and end states by drawing on a live camera image, which removes the ambiguity of phrases like 'near' or 'below'.
  • Long-horizon tasks become tractable because color-coded steps let the pipeline split one complex instruction into single-step subtasks.
  • Quiet environments such as libraries and hospitals become viable settings for robot instruction, since no speech is required.
  • Edge deployment becomes plausible because a 13-billion-parameter model fine-tuned on the 15K dataset can drive the whole pipeline.
  • Because the keypoint module tracks symbols rather than objects, performance degrades less in cluttered scenes with distractors than language-conditioned approaches do.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stricter test of the paper's separation claim would corrupt only the natural-language output while leaving keypoints and code intact; if success collapses, the language path is not as independent as claimed.
  • The same visual grammar could be extended with new primitives, for example a dashed line for 'wait here' or a cross for 'avoid', suggesting RoVI may be a general-purpose sketch language rather than a task-specific trick.
  • A user study that times sketching versus typing instructions and measures error rates would show whether the spatial-precision advantage survives use by non-experts.
  • Pairing RoVI with a short verbal goal, such as a sketch of where plus a word for what to do, could combine the convenience of speech with the precision of drawing; the paper leaves this hybrid unexplored.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes Robotic Visual Instruction (RoVI), a hand-drawn symbolic language for robot task specification that uses arrows, circles, colors, and numbers to encode spatial-temporal constraints. The authors also present VIEW, a pipeline that combines a VLM interpreting RoVI into language and Python code, a YOLOv8-based keypoint extractor, and a low-level keypoint-conditioned controller. They contribute a 15K-instance dataset (RoVI Book) for fine-tuning small VLMs, and report experiments across 11 tasks in real and simulated environments, claiming up to 87.5% success on unseen real-world tasks and outperforming language-conditioned baselines.

Significance. The paper addresses a real limitation of natural-language instruction for human-robot interaction—spatial ambiguity, verbosity, and inappropriateness in quiet settings—and proposes a concrete visual alternative with a user-friendly sketch interface. If the reported results are reliable, the work would be a useful step toward sketch-based HRI and provides a valuable open dataset as well as a plausible edge-deployment recipe via LoRA fine-tuning. The evaluation includes multi-step tasks, disturbances, and trajectory-following requirements, which are relevant stress cases. However, the central quantitative claim rests on a poorly evidenced dissociation between language comprehension and code execution, and the empirical evaluation has statistical and baseline-selection weaknesses. These issues are addressable but currently leave the headline claim under-supported.

major comments (4)
  1. [Section 6.3, Table 2 vs Table 1, Eq. (2)] The paper reports that VIEW-LLaVA-13B (RoVI Book) achieves 38% average task/planning comprehension (Table 2) yet 87.5% real-world execution success (Table 1). Since Eq. (2) defines ylanguage and ycode as outputs of the same fθ on the same input, low language comprehension should be expected to correlate with errors in the generated Python code. The explanation in Section 6.3 that "the execution function maps action and sequence errors, making it unaffected by perception errors" is not supported by any mechanism or per-trial evidence. The manuscript provides no examples of trials with incorrect language but correct code, no code-correctness metric, and no ablation that holds keypoints fixed while varying ycode. This is load-bearing because the 87.5% success rate is the paper's headline claim; without such evidence, it is unclear whether the VLM's code output contributes to execution or whether the keypoint module and low-level policy alone drive the results. Please provide per-trial annotations of language, code, and execution outcomes, a code-correctness evaluation, or an ablation that replaces or corrupts ycode.
  2. [Section 6.1, Table 1] All success rates are computed over 10 trials per task, with no error bars, confidence intervals, or significance tests. Several close comparisons (e.g., Task 4: VIEW-GPT4o 60% vs CoPa 60%; Task 7: VIEW-GPT4o 90% vs VIEW-LLaVA-13B 90%) cannot support the claim of "significant generalization capability" at this sample size. Please report per-trial data, confidence intervals, or statistical tests (e.g., exact binomial tests) for the headline comparisons.
  3. [Section 6.1, Section 6.2, Related Work] The evaluation omits existing sketch-based and visual-prompting methods such as RT-Sketch [45], MOKA [36], and ReKEP [28], despite citing them in Related Work. Since the paper's claim is that RoVI is a better visual instruction paradigm than natural language and goal images, the absence of a direct comparison against prior visual/sketch-conditioned policies makes the comparative conclusion incomplete. Please add at least one such baseline, or justify why the selected baselines suffice.
  4. [Section 6.2, Table 1 (Simulator)] The table footnote states that VIEW∗ denotes both VIEW-GPT4o and VIEW-LLaVA-13B "as their test results are identical." It is surprising that two different VLMs produce exactly identical results across all three simulator tasks. If the simulator tasks are essentially solved by the keypoint module and code templates independent of the VLM's comprehension, that supports the concern in the first major comment rather than the paper's interpretation. Please clarify how identical results arise and what this implies for the role of the VLM in VIEW.
minor comments (6)
  1. [Throughout] The name "LLaV A" appears with a spurious space in several places; it should be "LLaVA."
  2. [Table 2] The row labeled "Small Models" has all-zero scores but does not identify which models are included; this row should either be removed or broken down by model.
  3. [Figure 7] Figure 7 is referenced in the text but no detailed description is given, and the "detailed statistics" are deferred to supplementary material that is not provided with the manuscript; please include the relevant statistics in the main text or an appendix.
  4. [Section 6.3] The section mentions in-context learning but provides no details or examples of the in-context prompts; these should be included in the appendix so the evaluation is reproducible.
  5. [Section 5.3] The notation is inconsistent: v is defined as an image in R^{H×W×3} in Section 5.1 but Section 5.3 writes "v∈ R2 in pixel space"; please correct the variable or the dimension.
  6. [Section 5.4] The free parameter ε (keypoint reach threshold) is never specified; please report its value and, ideally, its sensitivity in the experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central 87.5% result is an empirical robot evaluation against external baselines, not a derivation whose output is assumed in its inputs.

full rationale

The paper's core claim is empirical: VIEW, using RoVI and keypoint extraction, achieves 87.5% success on unseen real-world tasks. This is measured against language-conditioned baselines (CoPa, VoxPoser) and VLA models (RT-1-X, Octo), so it does not reduce to a fitted parameter, a self-citation chain, or a by-construction equivalence. The keypoint module is trained separately, and the low-level policy is a standard cost-minimizing tracker; the fact that the RoVI arrow defines both the policy target and the trajectory-alignment metric is the intended operation of an instruction-following system, not a circular derivation. The RoVI Book dataset was generated by GPT-4o, but the comprehension benchmark in Table 2 is scored by human feedback on novel tasks, and GPT-4o is evaluated zero-shot, so this is a data-provenance concern rather than a by-construction equivalence. The one validity gap the paper itself acknowledges implicitly is the dissociation in Section 6.3 between 38% language comprehension and 87.5% execution; the claim that the execution function is unaffected by perception errors lacks per-trial code-vs-language evidence, but this is unsupported reasoning, not circular inference. No step in the claimed derivation chain is equivalent to its own input, so the circularity score is 0.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper introduces a new symbolic representation (RoVI) and a dataset, but no new physical entities, forces, or conserved quantities. RoVI is a language, not a posited entity with independent falsifiable consequences outside the paper.

free parameters (1)
  • epsilon keypoint reach threshold
    Used in Eq. 4 as the threshold Li(t) <= epsilon to mark a keypoint reached; no value is reported in the paper.
assumptions (5)
  • domain assumption 2D hand-drawn arrows and circles can encode sufficient spatial-temporal information for 3D manipulation.
    Core premise of RoVI; not proven, only demonstrated on author-designed tasks.
  • domain assumption Depth from an RGB-D camera provides reliable 3D coordinates for 2D keypoints.
    Used in Section 5.4 to map 2D keypoints to 3D; requires calibrated camera and visible surfaces.
  • ad hoc to paper The VLM-generated Python functions are correct even when the natural-language task/planning output is wrong.
    Needed to reconcile Table 2 (LLaVA-13B low comprehension) with Table 1 (high execution); no mechanism is provided.
  • domain assumption YOLOv8 keypoint module trained on 2k images generalizes to unseen RoVI drawings and environments.
    The keypoint module is trained on a small dataset and tested on unseen tasks; generalization is asserted but not formally established.
  • domain assumption AnyGrasp supplies a valid grasp pose for the starting keypoint.
    The pipeline relies on AnyGrasp to seed the initial pose; failure would break the execution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Robotic Visual Instruction." pith.science (2026). https://pith.science/paper/2IJT4P2K

@misc{pith2026250500693,
  author       = {Pith},
  title        = {Pith review of: Robotic Visual Instruction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2IJT4P2K}},
  note         = {Machine review of arXiv:2505.00693}
}
read the original abstract

Recently, natural language has been the primary medium for human-robot interaction. However, its inherent lack of spatial precision introduces challenges for robotic task definition such as ambiguity and verbosity. Moreover, in some public settings where quiet is required, such as libraries or hospitals, verbal communication with robots is inappropriate. To address these limitations, we introduce the Robotic Visual Instruction (RoVI), a novel paradigm to guide robotic tasks through an object-centric, hand-drawn symbolic representation. RoVI effectively encodes spatial-temporal information into human-interpretable visual instructions through 2D sketches, utilizing arrows, circles, colors, and numbers to direct 3D robotic manipulation. To enable robots to understand RoVI better and generate precise actions based on RoVI, we present Visual Instruction Embodied Workflow (VIEW), a pipeline formulated for RoVI-conditioned policies. This approach leverages Vision-Language Models (VLMs) to interpret RoVI inputs, decode spatial and temporal constraints from 2D pixel space via keypoint extraction, and then transform them into executable 3D action sequences. We additionally curate a specialized dataset of 15K instances to fine-tune small VLMs for edge deployment,enabling them to effectively learn RoVI capabilities. Our approach is rigorously validated across 11 novel tasks in both real and simulated environments, demonstrating significant generalization capability. Notably, VIEW achieves an 87.5% success rate in real-world scenarios involving unseen tasks that feature multi-step actions, with disturbances, and trajectory-following requirements. Project website: https://robotic-visual-instruction.github.io/

Figures

Figures reproduced from arXiv: 2505.00693 by the authors.

Figure 1
Figure 1. (Left) Robotic visual instruction is a hand-drawn approach for commanding robots, utilizing circles and arrows to convey task [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (Left) RoVI achieves an optimal balance of user-friendliness, interpretability, and spatiotemporal alignment. (Right) It shows [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. VIEW Architecture. This pipeline begins with a visual instruction drawn onto the initial observation. The VLM generates [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: This is an example to demonstrate the RoVI Book [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Robotic visual instruction is capable of generalizing to a variety of in-the-wild real-world situations, including multi-stage tasks [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: Performance comparison of average spatiotemporal [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 9
Figure 9. Figure 9: Error breakdown of language responses. Training with [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Showcase of two drawing styles in modified Open X [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spaces

    cs.CV 2025-05 reject novelty 6.0 of 10

    VeBrain unifies perception, spatial reasoning, and robot control in one MLLM by representing control as keypoint detection plus skill selection, with a robotic adapter for deployment.

Reference graph

Works this paper leans on

50 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [45]

    Rt-sketch: Goal-conditioned imitation learning from hand-drawn sketches

    Priya Sundaresan, Quan Vuong, Jiayuan Gu, Peng Xu, Ted Xiao, Sean Kirmani, Tianhe Yu, Michael Stark, Ajinkya Jain, Karol Hausman, et al. Rt-sketch: Goal-conditioned imitation learning from hand-drawn sketches. 2024. 2, 3

    work page 2024

  2. [36]

    Moka: Open-vocabulary robotic manipulation through mark-based visual prompting

    Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. arXiv preprint arXiv:2403.03174, 2024. 3

    arXiv 2024

  3. [28]

    Rekep: Spatio-temporal reasoning of rela- tional keypoint constraints for robotic manipulation

    Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of rela- tional keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652, 2024. 3, 4

    arXiv 2024

  4. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  5. [2]

    Gemini: A family of highly capable multimodal models

    Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 1, 2023. 7, 8

    arXiv 2023

  6. [3]

    Embodiedgpt: Vision-language pre-training via embodied chain of thought

    Anonymous. Embodiedgpt: Vision-language pre-training via embodied chain of thought. Under Review, 2023. 3

    work page 2023

  7. [4]

    The claude 3 model family: Opus, sonnet, haiku

    AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024. 7, 8

    work page 2024

  8. [5]

    Rt-h: Action hierarchies using language

    Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, De- bidatta Dwibedi, and Dorsa Sadigh. Rt-h: Action hierarchies using language. In https://arxiv.org/abs/2403.01823, 2024. 4

    arXiv 2024

Show all 50 references
  1. [6]

    Communication in human-robot interac- tion

    Andrea Bonarini. Communication in human-robot interac- tion. Current Robotics Reports, 1(4):279–285, 2020. 2

    2020

  2. [8]

    Rt-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakr- ishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022. 3

    2022 arXiv

  3. [10]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. 3

    2023 arXiv

  4. [11]

    Vip- llava: Making large multimodal models understand arbitrary visual prompts

    Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Vip- llava: Making large multimodal models understand arbitrary visual prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pag...

    2024

  5. [12]

    A survey on evaluation of large language models

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Tech- nology, 15(3):1–45, 2024. 2

    2024

  6. [13]

    Open X-Embodiment Collaboration, Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khaz- atsky, Ana...

  7. [14]

    Mechan- ical search: Multi-step retrieval of a target object occluded by clutter

    Michael Danielczuk, Andrey Kurenkov, Ashwin Balakr- ishna, Matthew Matl, David Wang, Roberto Martin-Martin, Animesh Garg, Silvio Savarese, and Ken Goldberg. Mechan- ical search: Multi-step retrieval of a target object occluded by clutter. In 2019 International Conference on Ro...

    2019

  8. [15]

    Keypoint action tokens enable in-context imitation learning in robotics

    Norman Di Palo and Edward Johns. Keypoint action tokens enable in-context imitation learning in robotics. In Proceed- ings of Robotics: Science and Systems (RSS), 2024. 3

    2024

  9. [16]

    Internlm-xcomposer2: Mastering free-form text- image composition and comprehension in vision-language large model

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Ji- aqi ...

    2024 arXiv

  10. [17]

    Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duck- worth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussai...

    2023 arXiv

  11. [18]

    Anygrasp: Robust and efficient grasp perception in spa- tial and temporal domains

    Hao-Shu Fang, Chenxi Wang, Hongjie Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, and Cewu Lu. Anygrasp: Robust and efficient grasp perception in spa- tial and temporal domains. IEEE Transactions on Robotics,

  12. [19]

    K-vil: Keypoints-based visual imitation learning

    Jianfeng Gao, Zhi Tao, No ´emie Jaquier, and Tamim As- four. K-vil: Keypoints-based visual imitation learning. IEEE Transactions on Robotics, 39(5):3888–3908, 2023. 3

    2023

  13. [20]

    Coda: Instructive chain-of-domain adaptation with severity-aware visual prompt tuning

    Ziyang Gong, Fuhao Li, Yupeng Deng, Deblina Bhattachar- jee, Xianzheng Ma, Xiangwei Zhu, and Zhenming Ji. Coda: Instructive chain-of-domain adaptation with severity-aware visual prompt tuning. In European Conference on Computer Vision, pages 130–148. Springer, 2024. 2

    2024

  14. [21]

    Rt-trajectory: Robotic task generalization via hindsight trajectory sketches

    Jiayuan Gu, Sean Kirmani, Paul Wohlhart, Yao Lu, Montser- rat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, et al. Rt-trajectory: Robotic task generalization via hindsight trajectory sketches. arXiv preprint arXiv:2311.01977, 2023. 2, 3

    2023 arXiv

  15. [22]

    Scaling up and distilling down: Language-guided robot skill acquisition

    Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Language-guided robot skill acquisition. In Proceedings of the 2023 Conference on Robot Learning ,

    2023

  16. [23]

    Visualizing natural lan- guage descriptions: A survey

    Kaveh Hassani and Won-Sook Lee. Visualizing natural lan- guage descriptions: A survey. ACM Computing Surveys (CSUR), 49(1):1–34, 2016. 2

    2016

  17. [24]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR, 2019. 2

    2019

  18. [25]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InIn- ternational Conference on Learning Representations , 2022. 5

    2022

  19. [26]

    Copa: General robotic manipulation through spatial constraints of parts with foundation models

    Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models. arXiv preprint arXiv:2403.08248, 2024. 3, 6

    2024 arXiv

  20. [27]

    V oxposer: Composable 3d value maps for robotic manipulation with language models

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023. 3, 4, 6

    2023 arXiv

  21. [29]

    Vima: General robot manip- ulation with multimodal prompts

    Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandku- mar, Yuke Zhu, and Linxi Fan. Vima: General robot manip- ulation with multimodal prompts. In Fortieth International Conference on Machine Learning, 2023. 3

    2023

  22. [30]

    Ultralytics YOLO, 2023

    Glenn Jocher, Jing Qiu, and Ayush Chaurasia. Ultralytics YOLO, 2023. 5, 8

    2023

  23. [31]

    Ananth Jonnavittula, Sagar Parekh, and Dylan P. Losey. View: Visual imitation learning with waypoints. arXiv preprint arXiv:2404.17906, 2024. 3

    2024 arXiv

  24. [32]

    Incorporating task progress knowledge for subgoal generation in robotic manipulation through image edits, 2024

    Xuhui Kang and Yen-Ling Kuo. Incorporating task progress knowledge for subgoal generation in robotic manipulation through image edits, 2024. 2, 3

    2024

  25. [33]

    Openvla: An open- source vision-language-action model, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Fos- ter, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kol- lar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Ope...

    2024

  26. [34]

    Affordance-guided reinforcement learning via visual prompting

    Olivia Y Lee, Annie Xie, Kuan Fang, Karl Pertsch, and Chelsea Finn. Affordance-guided reinforcement learning via visual prompting. arXiv preprint arXiv:2407.10341, 2024. 3

    2024 arXiv

  27. [35]

    Evaluating real-world robot manipulation policies in simulation

    Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating real-world robot manipulation policies in simulation. arXiv...

    2024 arXiv

  28. [37]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 5, 6, 7, 8

    2023

  29. [38]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 8

    2023 arXiv

  30. [39]

    Interactive language: Talking to robots in real time, 2022

    Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time, 2022. 2

    2022

  31. [40]

    Simple open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection. In European Conference on Computer Vision, pages 728–755. S...

    2022

  32. [41]

    Scaling open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. Advances in Neu- ral Information Processing Systems, 36, 2024. 8

    2024

  33. [42]

    Gen- erate subgoal images before act: Unlocking the chain-of- thought reasoning in diffusion model for robot manipulation with multimodal prompts

    Fei Ni, Jianye Hao, Shiguang Wu, Longxin Kou, Jiashun Liu, Yan Zheng, Wang Xian Bin, and Yuzheng Zhuang. Gen- erate subgoal images before act: Unlocking the chain-of- thought reasoning in diffusion model for robot manipulation with multimodal prompts. pages 13991–14000, 2024. 2

    2024

  34. [43]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo...

    2024

  35. [44]

    Zhao, and Chelsea Finn

    Lucy Xiaoyang Shi, Archit Sharma, Tony Z. Zhao, and Chelsea Finn. Waypoint-based imitation learning for robotic manipulation, 2023. 3

    2023

  36. [46]

    Large language models for robotics: Op- portunities, challenges, and perspectives

    Jiaqi Wang, Zihao Wu, Yiwei Li, Hanqi Jiang, Peng Shu, Enze Shi, Huawen Hu, Chong Ma, Yiheng Liu, Xuhui Wang, et al. Large language models for robotics: Op- portunities, challenges, and perspectives. arXiv preprint arXiv:2401.04334, 2024. 2

    2024 arXiv

  37. [47]

    Chang, Leonidas J

    Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated part-based interactive envi- ronment. In The IEEE Conference on Computer Vision and...

    2020

  38. [48]

    Flow as the cross-domain manipulation interface

    Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Wetzstein, Manuela Veloso, and Shuran Song. Flow as the cross-domain manipulation interface. In 8th Annual Confer- ence on Robot Learning, 2024. 2, 3

    2024

  39. [49]

    Set-of-mark prompting unleashes ex- traordinary visual grounding in gpt-4v, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes ex- traordinary visual grounding in gpt-4v, 2023. 6

    2023

  40. [50]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision)

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 9(1):1, 2023. 3

    2023 arXiv

  41. [51]

    3d-vla: A 3d vision-language-action generative world model

    Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d vision-language-action generative world model. arXiv preprint arXiv:2403.09631, 2024. 2, 3

    2024 arXiv

  42. [52]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 7

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.