Pith. sign in

REVIEW 3 major objections 5 minor 83 references

Today’s best vision-language models still miss nearly half of the modular decisions real robots must make, and those misses track real physical failure.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Expert-curated modular Robot-VQA benchmark of 474 questions across 39 robot tasks shows SOTA VLMs have large gaps that correlate with physical robot execution.

T0 review reviewed 2026-07-11 challenge →

load-bearing objection Useful modular VQA benchmark that fills a real gap for robot VLMs; physical correlations are suggestive but rest on a thin sample. the 3 major comments →

arxiv 2607.04610 v1 pith:ZAHYZQ7V submitted 2026-07-06 cs.RO

RoboVista: Evaluating Vision Language Models for Diverse Robot Applications

Shuangyu Xie , Kaiyuan Chen , Ziyang Chen , Simeon Adebola , Yixuan Huang , Zehan Ma , Tianshuang Qiu , Wentao Yuan
show 3 more authors
Dhruv Shah Pannag R. Sanketi Ken Goldberg
This is my paper
classification cs.RO
keywords vision-language modelsrobot question answeringRoboVistamodular roboticsvisual question answeringembodied reasoningsurgical roboticsbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Robots in factories, farms, homes, and operating rooms do not act as one big black box; they chain many small, modular decisions—what is visible, what to grasp next, whether a motion is safe, how to recover. This paper claims that if vision-language models are to become the common reasoning layer for such systems, we must test those modular decisions directly rather than only scoring end-to-end teleoperated trajectories. The authors therefore introduce Robot Question Answering (RQA), a way to turn real robot pipelines into expert-verified multiple-choice visual questions, and release RoboVista, 474 such questions spanning 39 task types and six application domains. On this benchmark even the strongest models top out near 56 percent accuracy, and the same models’ RoboVista scores strongly predict how well they estimate distances and guide surgical knot-tying on physical hardware. A sympathetic reader cares because the gap is now measurable, domain-by-domain, and because the physical correlation suggests that fixing RoboVista failures would improve actual robots.

Core claim

State-of-the-art vision-language models exhibit large, persistent performance gaps on modular robot decision points (best overall accuracy 56.5 percent), and a model’s RoboVista score correlates strongly with real-world spatial estimation error and with progress on closed-loop surgical knot-tying under shared autonomy.

What carries the argument

Robot Question Answering (RQA): a module-level abstraction that maps each functional block of a robot pipeline—perception, high-level decision making, motion awareness, failure recovery—into a robot-centric visual question with a single verified answer and human rationale, then assembles those questions into the RoboVista benchmark.

Load-bearing premise

The 474 expert-built multiple-choice questions, each tied to one modular decision taken from published robot systems, are representative enough of real decision quality that accuracy on them can stand in for how well a model would help an actual robot.

What would settle it

Run the same physical bimanual distance-alignment and dVRK knot-tying protocols with a model whose RoboVista score has been deliberately raised (or lowered) by training or prompting, and check whether physical error and task progress move in the predicted direction; if the correlation collapses, the central claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Robot Question Answering (RQA), a modular framework that maps decision points from classical robot pipelines (perception, high-level planning, motion awareness, failure recovery) into robot-centric multiple-choice VQA instances, and instantiates it as RoboVista: 474 expert-annotated questions spanning 39 task types across agriculture, industry, domestic, surgical, driving, and open datasets. Each instance includes robot-visible or onboard imagery, five answer choices, and human rationales. Zero-shot evaluation of open- and closed-source VLMs shows substantial gaps (best overall accuracy 56.5% for Gemini 2.5 Pro). Ablations examine Chain-of-Thought and in-context learning; failure analysis separates perception from reasoning errors. Two physical experiments (bimanual gripper alignment and shared-autonomy dVRK knot-tying) report correlations between RoboVista scores and real-world spatial error / task progress.

Significance. If the reported gaps and the proxy relationship hold, RoboVista supplies a needed diagnostic complement to trajectory-scale embodied VQA benchmarks (Robo2VLM, RoboBrain, ERQA). Its deliberate coverage of data-scarce modular domains (surgery, agriculture, industrial deformable assembly) and expert-verified rationales are genuine strengths. The physical correlation experiments, even if under-powered, are a welcome attempt to link VQA accuracy to closed-loop execution rather than treating the benchmark as self-justifying. The work is therefore of clear interest to the robotics and VLM communities as both an evaluation resource and a design template for modular robot-centric VQA.

major comments (3)
  1. Section VI and Fig. 7b: The claimed strong negative correlation between RoboVista score and bimanual position error (Pearson r = −0.78, Spearman ρ = −0.93) rests on roughly 6–7 models that already appear in the main zero-shot ranking. No p-values, confidence intervals, or leave-one-out sensitivity are reported. With such small N a single outlier can dominate; the manuscript should either enlarge the model set, report statistical significance, or explicitly frame the result as exploratory rather than confirmatory of proxy validity.
  2. Table IV (surgical knot-tying): Only three models are evaluated, and the 16 closed-loop queries are drawn from the same surgical domain (and RQA construction process) that contributes to the RoboVista-Surgery score. This partial non-independence weakens the claim that higher benchmark accuracy predicts greater real-task progress. The paper should either add held-out procedural stages / models or qualify the association more carefully.
  3. Section IV Quality Control and Appendix A: Inter-annotator agreement (e.g., exact-match or rationale consistency) is not quantified, nor is the procedure for generating and validating the four distractors described beyond “algorithmically grounded.” Because the central claim treats accuracy on these 474 items as a reliable proxy for modular decision quality, a short quantitative reliability analysis (or explicit statement of its absence) is load-bearing and should be added.
minor comments (5)
  1. Table I vs. Table II: GPT-5 overall accuracy is listed as 48.1% (zero-shot) in Table I but 55.5% in Table II; clarify whether different checkpoints, decoding, or subset filtering explain the discrepancy.
  2. Fig. 4 table header says “39 unique task types” while Appendix Table V reports “Unique Tasks 33”; reconcile the counts.
  3. Section V decoding: main text states temperature 0.7; Appendix B states temperature 0.0 (greedy). Align the reported protocol.
  4. Figs. 5 and 6 (and Appendix Figs. 8–9) appear to reuse nearly identical failure-analysis diagrams with swapped model names; ensure captions and percentages match the intended model.
  5. Abstract and Introduction claim “strong correlation”; given the sample-size caveats above, softer language (“suggestive association”) would better match the evidence presented.

Circularity Check

1 steps flagged

No derivation circularity; mild non-independence only in the surgical physical validation, where progress is itself sequential VQA accuracy on RQA questions from the same domain.

specific steps
  1. other [Section VI.B / Table IV / Appendix Table VIII]
    "Across all models, higher RoboVista-surgical scores are associated with greater shared autonomy task progress. Models with stronger benchmark performance consistently complete more stages of the knot-tying procedure before requiring assistance. ... Human experts ... design a sequence of VQA questions with RQA framework that correspond to critical decision points in the knot-tying process ... We evaluate multiple vision-language models by asking the designed questions in sequence and measuring how far the knot-tying task can progress"

    Task progress is operationalized as the number of sequential RQA VQAs answered correctly (with ≤3 interventions) on a 16-question knot-tying sequence whose construction, visual style, and decision points are identical to those used for the 30 Knot-Tying items that contribute to the RoboVista-Surgery score. The association between RoboVista-Surgery accuracy and this progress metric is therefore partly by construction rather than an independent physical measurement of continuous robot execution.

full rationale

RoboVista is an empirical VQA benchmark constructed from external robot systems, papers, and fresh expert annotations; model accuracies are measured against fixed human labels with no free parameters fitted to produce the reported gaps (best 56.5 %). The bi-manual gripper experiment measures continuous distance/position errors under geometric priors and reports an independent correlation (Pearson r = −0.78) with overall RoboVista scores; that measurement does not reduce to the VQA labels by construction. The sole mild circularity is the surgical knot-tying closed-loop experiment: progress is defined by sequential correctness on 16 RQA-designed VQAs that match the style, domain, and source of the 30 Knot-Tying items already inside RoboVista-Surgery, so the reported association is partly expected rather than fully independent physical evidence. No self-definitional equations, fitted-then-predicted quantities, uniqueness theorems, or ansatz smuggling appear. Self-citations are ordinary source attributions for the underlying robot pipelines and do not load-bear any uniqueness claim. Score 2 reflects one non-central, partial non-independence; the central empirical claims remain self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 2 invented entities

The paper is an empirical benchmark construction and evaluation study. It inherits standard robotics modular-pipeline abstractions and the VQA evaluation paradigm; no free parameters are fitted to produce the headline accuracy gaps or correlations. The only invented constructs are the RQA mapping itself and the RoboVista collection.

axioms (3)
  • domain assumption Complex robot behaviors can be usefully decomposed into the four functional layers of perception, high-level decision making, motion/action estimation and failure recovery (Fig. 2 and Section III-A).
    Standard modular robotics design assumption drawn from classical pipelines; used to structure every RQA instance.
  • domain assumption A multiple-choice VQA instance with five options and an expert rationale is a faithful probe of the underlying module decision (Definition 1–3).
    Inherited from the broader VQA literature and asserted as the evaluation interface.
  • domain assumption Expert annotators (graduate-level and above, majority PhDs in robotics) produce unambiguous, visually grounded ground-truth answers.
    Quality-control claim in Section IV; load-bearing for treating accuracy as meaningful.
invented entities (2)
  • Robot Question Answering (RQA) framework no independent evidence
    purpose: Structured mapping from a module tuple (E,X,U,C) to a Robot-VQA 5-tuple.
    New formalization introduced in Section III; no independent evidence outside the paper’s own construction.
  • RoboVista benchmark no independent evidence
    purpose: 474 expert-annotated multiple-choice questions spanning 39 task types and six domains.
    The concrete dataset whose statistics and results constitute the paper’s main empirical contribution.

reviewed 2026-07-11 · how reviews work

0 comments
Cite this review

Pith. "Pith review of RoboVista: Evaluating Vision Language Models for Diverse Robot Applications." pith.science (2026). https://pith.science/paper/ZAHYZQ7V

@misc{pith2026260704610,
  author       = {Pith},
  title        = {Pith review of: RoboVista: Evaluating Vision Language Models for Diverse Robot Applications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZAHYZQ7V}},
  note         = {Machine review of arXiv:2607.04610}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Diverse applications for robotics, such as industry and agriculture, require robots to operate across various embodiments, changing visual conditions, and complex planning. Vision-Language Models (VLMs) offer a promising foundation for general-purpose and interpretable robotic reasoning. Aligning VLMs with diverse robot applications requires a modular understanding of the individual decision components that underlie robotic behavior. Capturing such structure is challenging for conventional robot benchmarks that are primarily based on teleoperated, end-to-end datasets. We propose Robot Question Answering (RQA), a modular evaluation framework and RoboVista, a benchmark curated from real robotic systems, research papers, and expert annotations. RoboVista contains 474 Visual Question Answering (VQA) instances with human annotated reasoning and covers 39 unique task types in agricultural, industrial, domestic, surgical robotics, autonomous driving, and open robot datasets. Experiments on RoboVista show that state-of-the-art VLMs exhibit substantial gaps. Physical robot experiments suggest strong correlation between RoboVista performance and real-world task execution.

Figures

Figures reproduced from arXiv: 2607.04610 by Dhruv Shah, Kaiyuan Chen, Ken Goldberg, Pannag R. Sanketi, Shuangyu Xie, Simeon Adebola, Tianshuang Qiu, Wentao Yuan, Yixuan Huang, Zehan Ma, Ziyang Chen.

Figure 1
Figure 1. Figure 1: RoboVista Overview. To support future robot applications, RoboVista presents fine-grained spatial understanding and embodied decision-making challenges for Vision–Language Models (VLMs). Grounded in 6 robot application domains and 39 diverse tasks, RoboVista is an expert-annotated Visual Question Answering (VQA) dataset emphasizing variable robot embodiments (left), interactions with deformable objects and… view at source ↗
Figure 2
Figure 2. Figure 2: Module abstraction of diverse robot applications pipelines. The figure illustrates a modular view of real-world robot systems, spanning diverse application scenes and robot embodiments. Robot operation can be decomposed into four functional layers in perception, high-level decision making, motion / action estimation and failure recovery. visual observations and paired with detailed human reasoning explanat… view at source ↗
Figure 3
Figure 3. Figure 3: RQA case studies across domains. This figure provide two example on how the RQA framework maps continuous physical robot states into modular, discrete VQA pairs that correspond to critical real-world decision points, such as tool alignment and tension management. Top: Ambidextrous bin picking with Dex-Net [59] Bottom: Surgical knot tying [5]. In both cases, robotic decision points are decomposed into perce… view at source ↗
Figure 4
Figure 4. Figure 4: RoboVista Gallery. Representative visual observations and tasks across the six domains, including agriculture, autonomous driving, domestic, industrial, surgical, and open robot datasets. The table summarizes task types and the distribution of perception- and planning-centric questions. TABLE I: Zero-Shot Performance of Multimodal Foundation Models on RQA (%). Model All Agriculture Driving Home Industry Su… view at source ↗
Figure 5
Figure 5. Figure 5: Failure Analysis of Qwen2.5-VL 7B. We analyze the reasoning chain of Qwen2.5-VL 7B and categorize failures in misidentification and reasoning. Total 474 Correct 230 (48.5%) Wrong 244 (51.5%) Sound Reasoning 218 (46.0%) Flawed Reasoning 12 (2.5%) Visual Perception 117 (24.7%) Reasoning 103 (21.7%) Spatial Reasoning 22 (4.6%) Other 2 (0.4%) Misidentification 96 (20.3%) Spatial 46 (9.7%) Counting 8 (1.7%) Tas… view at source ↗
Figure 6
Figure 6. Figure 6: Failure Analysis of Qwen3-VL 235B. We analyze the reasoning chain of Qwen3-VL 235B and categorize failures in misidentification and reasoning. increases calibration error across all models, with absolute CE increases of up to 9.7%. This trend indicates that ICL encourages models to produce more confident but less reliable predictions, which could amplify hallucinated reasoning when visual evidence is ambig… view at source ↗
Figure 7
Figure 7. Figure 7: Bi-manual gripper alignment task Setup and correlation with RoboVista performance. The VLM estimates the distance between gripper tips and plans motions based on spatial priors. Higher RoboVista scores strongly correlates to lower estimation and execution errors. as misunderstanding which object is being manipulated or misinterpreting contact and interaction states. Failure Analysis on Qwen3-235B-A22B . We… view at source ↗
Figure 8
Figure 8. Figure 8: Failure Analysis of Qwen2.5-VL 7B. We analyze the reasoning chain of Qwen2.5-VL 7B and categorize failures in misidentification and reasoning. the test question itself) using a fixed seed for reproducibility. The prompt is structured as: === EXAMPLE === [Image 1: Example image] Question: {example_question} Options: {example_choices} Answer: {correct_answer} === YOUR QUESTION === [Image 2: Your image] {test… view at source ↗
Figure 10
Figure 10. Figure 10: Evaluation Setup of Distance Estimation. From top to bottom is kitchen, industrial, empty, checkerboard, and plants. DISTANCE: <value> cm COMMAND: [("<direction>", <amount>, "cm")] where each command tuple specifies a direction (forward, backward, left, right, up, down) and a metric distance. The prompt explicitly instructs models to leverage visible back￾ground objects as geometric priors for scale estim… view at source ↗
Figure 11
Figure 11. Figure 11: Visual Questions for surgical knot tying. The system prompt establishes task context: “You are an expert surgical assistant AI monitoring an emulated robotic surgery task on the da Vinci Research Kit robot. The ex￾perimental scene contains two grippers, a surgical needle, a suture thread, and a wound phantom. Your task is to monitor and guide a teleoperated surgical knot tying procedure.” The VLM is queri… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

83 extracted references · 17 linked inside Pith

  1. [1]

    Industrial robot grasping with deep learning using a programmable logic controller (plc),

    E. Solowjow, I. Ugalde, Y . Shahapurkar, J. Aparicio, J. Mahler, V . Satish, K. Goldberg, and H. Claussen, “Industrial robot grasping with deep learning using a programmable logic controller (plc),” in2020 IEEE 16th International Confer- ence on Automation Science and Engineering (CASE), 2020, pp. 97–103

    2020

  2. [2]

    Automating deformable gasket assembly,

    S. Adebola, T. Sadjadpour, K. El-Refai, W. Panitch, Z. Ma, R. Lin, T. Qiu, S. Ganti, C. Le, J. Drake, et al., “Automating deformable gasket assembly,” in2024 IEEE 20th International Conference on Automation Science and Engineering (CASE), IEEE, 2024, pp. 4146–4153

    2024

  3. [3]

    Energy efficient planning for repetitive heterogeneous tasks in precision agriculture,

    S. Xie, K. Goldberg, and D. Song, “Energy efficient planning for repetitive heterogeneous tasks in precision agriculture,” in2025 IEEE International Conference on Robotics and Automation (ICRA), 2025, pp. 7139–7145

    2025

  4. [4]

    Can machines garden? systematically comparing the alphagarden vs. professional horticulturalists,

    S. Adebola, R. Parikh, M. Presten, S. Sharma, S. Aeron, A. Rao, S. Mukherjee, T. Qu, C. Wistrom, E. Solowjow, et al., “Can machines garden? systematically comparing the alphagarden vs. professional horticulturalists,” in2023 IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 11 779–11 785

    2023

  5. [5]

    Surgical d-knot: Augmented dexterity for tying double knots by monitoring optical flow in monocular attention windows,

    Z. Chen, K. Hari, T. Dasari, K. Shieh, R. Jain, D. M. Fer, G. Guthart, and K. Goldberg, “Surgical d-knot: Augmented dexterity for tying double knots by monitoring optical flow in monocular attention windows,” in2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2025, pp. 2148–2155

    2025

  6. [6]

    Stitch 2.0: Extending augmented suturing with ekf needle estimation and thread management,

    K. Hari, Z. Chen, H. Kim, and K. Goldberg, “Stitch 2.0: Extending augmented suturing with ekf needle estimation and thread management,”IEEE Robotics and Automation Letters, vol. 10, no. 12, pp. 12 700–12 707, 2025

    2025

  7. [7]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning transferable visual models from natural language supervision,” inProceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang, Eds., ser. Proceedings of Machine Learning Research, vol. 139, PMLR, 18...

    2021

  8. [8]

    An introduction to vision-language modeling,

    F. Bordes, R. Y . Pang, A. Ajay, A. C. Li, A. Bardes, S. Petryk, O. Ma ˜nas, Z. Lin, A. Mahmoud, B. Jayaraman, et al., “An introduction to vision-language modeling,”arXiv preprint arXiv:2405.17247, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Design2Code: Benchmarking multimodal code generation for automated front-end engineering,

    C. Si, Y . Zhang, R. Li, Z. Yang, R. Liu, and D. Yang, “Design2Code: Benchmarking multimodal code generation for automated front-end engineering,” inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), L. Chiruzzo, A. Ritter, and L. ...

    2025

  10. [10]

    G. R. Team, S. Abeyruwan, et al.,Gemini robotics: Bringing ai into the physical world, 2025. arXiv: 2503.20020[cs.RO]

    Pith/arXiv arXiv 2025

  11. [11]

    V oxposer: Composable 3d value maps for robotic manipulation with language models,

    W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei, “V oxposer: Composable 3d value maps for robotic manipulation with language models,”arXiv preprint arXiv:2307.05973, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Prompting with the future: Open-world model predictive control with interactive digital twins,

    C. Ning, K. Fang, and W. -C. Ma, “Prompting with the future: Open-world model predictive control with interactive digital twins,” inProceedings of Robotics: Science and Systems (RSS), 2025

    2025

  13. [13]

    Vqa: Visual question answering,

    S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, “Vqa: Visual question answering,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2425–2433

    2015

  14. [14]

    Visual question answering: A survey of methods, datasets, evaluation, and challenges,

    B. S. Kim, J. Kim, D. Lee, and B. Jang, “Visual question answering: A survey of methods, datasets, evaluation, and challenges,”ACM Comput. Surv., vol. 57, no. 10, May 2025

    2025

  15. [15]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering,

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 6904–6913

    2017

  16. [16]

    What is right for me is not yet right for you: A dataset for grounding relative directions via multi-task learning,

    J. H. Lee, M. Kerzel, K. Ahrens, C. Weber, and S. Wermter, “What is right for me is not yet right for you: A dataset for grounding relative directions via multi-task learning,”arXiv preprint arXiv:2205.02671, 2022

    Pith/arXiv arXiv 2022

  17. [17]

    Mmmu: A massive multi- discipline multimodal understanding and reasoning benchmark for expert agi,

    X. Yue, Y . Ni, T. Zheng, K. Zhang, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sun, et al., “Mmmu: A massive multi- discipline multimodal understanding and reasoning benchmark for expert agi,” in2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 9556–9567

    2024

  18. [18]

    MMMU-pro: A more robust multi-discipline multimodal understanding benchmark,

    X. Yue, T. Zheng, Y . Ni, Y . Wang, K. Zhang, S. Tong, Y . Sun, B. Yu, G. Zhang, H. Sun, et al., “MMMU-pro: A more robust multi-discipline multimodal understanding benchmark,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds., Vienna, ...

    2025

  19. [19]

    Robo2vlm: Improving visual question answering using large- 10 scale robot manipulation data,

    K. Chen, S. Xie, Z. Ma, P. R. Sanketi, and K. Goldberg, “Robo2vlm: Improving visual question answering using large- 10 scale robot manipulation data,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025

    2025

  20. [20]

    Robobrain: A unified brain model for robotic manipulation from abstract to concrete,

    Y . Ji, H. Tan, J. Shi, X. Hao, Y . Zhang, H. Zhang, P. Wang, M. Zhao, Y . Mu, P. An, et al., “Robobrain: A unified brain model for robotic manipulation from abstract to concrete,” CVPR, 2025

    2025

  21. [21]

    O. X.-E. Collaboration, A. O’Neill, A. Rehman, A. Gupta, A. Maddukuri, A. Gupta, A. Padalkar, et al.,Open X-Embodiment: Robotic learning datasets and RT-X models, https://arxiv.org/ abs/2310.08864, 2023

    Pith/arXiv arXiv 2023

  22. [22]

    Manip: A modular architecture for integrating inter- active perception for robot manipulation,

    J. Yu, T. Sadjadpour, A. O’Neill, M. Khfifi, L. Y . Chen, R. Cheng, M. Z. Irshad, A. Balakrishna, T. Kollar, and K. Goldberg, “Manip: A modular architecture for integrating inter- active perception for robot manipulation,” in2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2024, pp. 1283–1289

    2024

  23. [23]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou, et al., “Chain-of-thought prompting elicits reasoning in large language models,”Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  24. [24]

    A survey on in-context learning,

    Q. Dong, L. Li, D. Dai, C. Zheng, J. Ma, R. Li, H. Xia, J. Xu, Z. Wu, B. Chang, et al., “A survey on in-context learning,” in Proceedings of the 2024 conference on empirical methods in natural language processing, 2024, pp. 1107–1128

    2024

  25. [25]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,”Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  26. [26]

    Z. Chen, K. Hari, and K. Goldberg,Surgical debridement dataset, https://github.com/open-h-embodiment/data-collection, GitHub repository, 2025

    2025

  27. [27]

    Toward robotic weed control: Detection of nutsedge weed in bermuda- grass turf using inaccurate and insufficient training data,

    S. Xie, C. Hu, M. Bagavathiannan, and D. Song, “Toward robotic weed control: Detection of nutsedge weed in bermuda- grass turf using inaccurate and insufficient training data,”IEEE Robotics and Automation Letters, vol. 6, no. 4, pp. 7365–7372, 2021

    2021

  28. [28]

    Coupled active perception and manipulation planning for a mobile manipulator in precision agriculture applications,

    S. Xie, C. Hu, D. Wang, J. Johnson, M. Bagavathiannan, and D. Song, “Coupled active perception and manipulation planning for a mobile manipulator in precision agriculture applications,” inIEEE International Conference on Robotics and Automation (ICRA), 2024

    2024

  29. [29]

    Botany-bot: Digital twin monitoring of occluded and underleaf plant structures with gaussian splats,

    S. Adebola, C. M. Kim, J. Kerr, S. Xie, P. Akella, J. L. S. Rincon, E. Solowjow, and K. Goldberg, “Botany-bot: Digital twin monitoring of occluded and underleaf plant structures with gaussian splats,” in2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2025, pp. 1839– 1846

    2025

  30. [30]

    Omni-scan: Creating visually-accurate digital twin object models using a bimanual robot with handover and gaussian splat merging,

    T. Qiu, Z. Ma, K. El-Refai, H. Shah, C. M. Kim, J. Kerr, and K. Goldberg, “Omni-scan: Creating visually-accurate digital twin object models using a bimanual robot with handover and gaussian splat merging,” in2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2025, pp. 18 782–18 789

    2025

  31. [31]

    The teenager’s problem: Efficient garment decluttering as probabilistic set cover,

    A. Adler, A. Ahmad, Y . Qiu, S. Wang, W. C. Agboh, E. Llontop, T. Qiu, J. Ichnowski, T. Kollar, R. Cheng, et al., “The teenager’s problem: Efficient garment decluttering as probabilistic set cover,”arXiv preprint arXiv:2310.16951, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,

    B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia, “Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2024, pp. 14 455–14 465

    2024

  33. [33]

    Karamcheti, S

    S. Karamcheti, S. Nair, A. Balakrishna, P. Liang, T. Kollar, and D. Sadigh,Prismatic vlms: Investigating the design space of visually-conditioned language models, 2024. arXiv: 2402.07865 [cs.CV]

    Pith/arXiv arXiv 2024

  34. [34]

    Openvla: An open-source vision-language-action model,

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al., “Openvla: An open-source vision-language-action model,” in Proceedings of the Conference on Robot Learning (CoRL), 2024

    2024

  35. [35]

    π0: A generalist robot policy,

    P. I. Team, “ π0: A generalist robot policy,”arXiv preprint arXiv:2410.24164, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Text2motion: From natural language instructions to feasible plans,

    K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg, “Text2motion: From natural language instructions to feasible plans,”Autonomous Robots, vol. 47, pp. 1345–1365, 2023

    2023

  37. [37]

    PaLM- E: An embodied multimodal language model,

    D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al., “PaLM- E: An embodied multimodal language model,” inProceedings of the 40th International Conference on Machine Learning (ICML), 2023, pp. 8469–8488

    2023

  38. [38]

    Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation,

    W. Huang, C. Wang, Y . Li, R. Zhang, and L. Fei-Fei, “Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation,”arXiv preprint arXiv:2409.01652, 2024

    Pith/arXiv arXiv 2024

  39. [39]

    Moka: Open-world robotic manipulation through mark-based visual prompting,

    K. Fang, F. Liu, P. Abbeel, and S. Levine, “Moka: Open-world robotic manipulation through mark-based visual prompting,” Robotics: Science and Systems XX, 2024

    2024

  40. [40]

    Kalie: Fine-tuning vision-language models for open-world manipulation without robot data,

    G. Tang, S. Rajkumar, Y . Zhou, H. R. Walke, S. Levine, and K. Fang, “Kalie: Fine-tuning vision-language models for open-world manipulation without robot data,” in2025 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2025, pp. 9507–9515

    2025

  41. [41]

    Maestro: Orchestrating robotics modules with vision-language models for zero-shot generalist robots,

    J. Shi, R. Yang, K. Chao, B. S. Wan, Y . S. Shao, J. Lei, J. Qian, L. Le, P. Chaudhari, K. Daniilidis, et al., “Maestro: Orchestrating robotics modules with vision-language models for zero-shot generalist robots,” inNeurIPS 2025 Workshop on Space in Vision, Language, and Embodied AI, 2025

    2025

  42. [42]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning,

    A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque, “ChartQA: A benchmark for question answering about charts with visual and logical reasoning,” inFindings of the Associa- tion for Computational Linguistics: ACL 2022, S. Muresan, P. Nakov, and A. Villavicencio, Eds., Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 2263–2279

    2022

  43. [43]

    P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao,Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts,

  44. [44]

    arXiv: 2310.02255[cs.CV]

    Pith/arXiv arXiv

  45. [45]

    Embodied question answering,

    A. Das, S. Datta, G. Gkioxari, S. Lee, D. Parikh, and D. Batra, “Embodied question answering,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018

    2018

  46. [46]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,

    P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, S. Gould, and A. van den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018

    2018

  47. [47]

    Manipbench: Benchmarking vision-language models for low-level robot manipulation,

    E. Zhao, V . Raval, H. Zhang, J. Mao, Z. Shangguan, S. Nikolaidis, Y . Wang, and D. Seita, “Manipbench: Benchmarking vision-language models for low-level robot manipulation,” Conference on Robot Learning, 2025

    2025

  48. [48]

    Robomind: Benchmark on multi- embodiment intelligence normative data for robot manipulation,

    K. Wu, C. Hou, J. Liu, Z. Che, X. Ju, Z. Yang, M. Li, Y . Zhao, Z. Xu, G. Yang, et al., “Robomind: Benchmark on multi- embodiment intelligence normative data for robot manipulation,” inRobotics: Science and Systems (RSS) 2025, Robotics: Science and Systems Foundation, 2025

    2025

  49. [49]

    TurnaboutLLM: A deductive reasoning benchmark from detective games,

    Y . Yuan, M. He, M. A. Shahid, Z. Li, J. Huang, and L. Zhang, “TurnaboutLLM: A deductive reasoning benchmark from detective games,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V . Peng, Eds., Suzhou, China: Association for Computational Linguistics, Nov. 2025...

    2025

  50. [50]

    MuSR: Testing the limits of chain-of-thought with multistep 11 soft reasoning,

    Z. R. Sprague, X. Ye, K. Bostrom, S. Chaudhuri, and G. Durrett, “MuSR: Testing the limits of chain-of-thought with multistep 11 soft reasoning,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  51. [51]

    Glazer, E

    E. Glazer, E. Erdil, T. Besiroglu, D. Chicharro, E. Chen, A. Gunning, C. F. Olsson, J. -S. Denain, A. Ho, E. de Oliveira Santos, et al.,Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai, 2025. arXiv: 2411. 04872[cs.AI]

    2025

  52. [52]

    Siciliano and O

    B. Siciliano and O. Khatib,Handbook of Robotics. Springer, 2016, Part C: Sensing and Perception, pp. 525-894

    2016

  53. [53]

    S. M. LaValle,Planning Algorithms. Cambridge, UK: Cam- bridge University Press, 2006

    2006

  54. [54]

    K. M. Lynch and F. C. Park,Modern Robotics: Mechanics, Planning, and Control, 1st. USA: Cambridge University Press, 2017

    2017

  55. [55]

    A unified approach for motion and force control of robot manipulators: The operational space formulation,

    O. Khatib, “A unified approach for motion and force control of robot manipulators: The operational space formulation,”IEEE Journal on Robotics and Automation, vol. 3, no. 1, pp. 43–53, 1987

    1987

  56. [56]

    Neural module networks,

    J. Andreas, M. Rohrbach, T. Darrell, and D. Klein, “Neural module networks,” in2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 39–48

    2016

  57. [57]

    Spatial planning: A configuration space approach,

    T. Lozano-P ´erez, “Spatial planning: A configuration space approach,”IEEE Transactions on Computers, no. 2, pp. 108– 120, 1983

    1983

  58. [58]

    Thrun, W

    S. Thrun, W. Burgard, and D. Fox,Probabilistic Robotics. MIT Press, 2005

    2005

  59. [59]

    Chomp: Gradient optimization techniques for efficient motion planning,

    N. Ratliff, M. Zucker, J. A. Bagnell, and S. Srinivasa, “Chomp: Gradient optimization techniques for efficient motion planning,” in2009 IEEE International Conference on Robotics and Automation, IEEE, 2009, pp. 489–494

    2009

  60. [60]

    Learning ambidextrous robot grasping policies,

    J. Mahler, M. Matl, V . Satish, M. Danielczuk, B. DeRose, S. McKinley, and K. Goldberg, “Learning ambidextrous robot grasping policies,”Science Robotics, vol. 4, no. 26, eaau4984, 2019

    2019

  61. [61]

    Automated pruning of polyculture plants,

    M. Presten, R. Parikh, S. Aeron, S. Mukherjee, S. Adebola, S. Sharma, M. Theis, W. Teitelbaum, and K. Goldberg, “Automated pruning of polyculture plants,” in2022 IEEE 18th International Conference on Automation Science and Engineering (CASE), IEEE, 2022, pp. 242–249

    2022

  62. [62]

    Algorithm and system development for robotic micro-volume herbicide spray towards precision weed management,

    C. Hu, S. Xie, D. Song, J. A. Thomasson, R. G. H. IV, and M. Bagavathiannan, “Algorithm and system development for robotic micro-volume herbicide spray towards precision weed management,”IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 11 633–11 640, 2022

    2022

  63. [63]

    Toward precise robotic weed flaming using a mobile manipulator with a flamethrower,

    D. Wang, C. Hu, S. Xie, J. Johnson, H. Ji, Y . Jiang, M. Bagavathiannan, and D. Song, “Toward precise robotic weed flaming using a mobile manipulator with a flamethrower,”arXiv preprint arXiv:2407.04929, 2024

    Pith/arXiv arXiv 2024

  64. [64]

    GM/SAE AutoDrive Challenge I (Year 4),Gm/sae autodrive challenge i (year 4) yearbook, https://online.flippingbook.com/ view/101355423/, Online flippingbook yearbook for the Year 4 competition of the AutoDrive Challenge, 2021

    arXiv 2021

  65. [65]

    Points2plans: From point clouds to long-horizon plans with composable relational dynamics,

    Y . Huang, C. Agia, J. Wu, T. Hermans, and J. Bohg, “Points2plans: From point clouds to long-horizon plans with composable relational dynamics,” in2025 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2025, pp. 1208–1216

    2025

  66. [66]

    Fail2progress: Learning from real-world robot failures with stein variational inference,

    Y . Huang, N. Alvina, M. Devendran Shanthi, and T. Hermans, “Fail2progress: Learning from real-world robot failures with stein variational inference,” inConference on Robot Learning (CoRL), 2025, 2025

    2025

  67. [67]

    Handloom: Learned tracing of one-dimensional objects for inspection and manipulation,

    V . Viswanath, K. Shivakumar, M. Parulekar, J. Ajmera, J. Kerr, J. Ichnowski, R. Cheng, T. Kollar, and K. Goldberg, “Handloom: Learned tracing of one-dimensional objects for inspection and manipulation,” inConference on Robot Learning, PMLR, 2023, pp. 341–357

    2023

  68. [68]

    Bagging by learning to singulate layers using interactive perception,

    L. Y . Chen, B. Shi, R. Lin, D. Seita, A. Ahmad, R. Cheng, T. Kollar, D. Held, and K. Goldberg, “Bagging by learning to singulate layers using interactive perception,” in2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2023, pp. 3176–3183

    2023

  69. [69]

    Autobag: Learning to open plastic bags and insert objects,

    L. Y . Chen, B. Shi, D. Seita, R. Cheng, T. Kollar, D. Held, and K. Goldberg, “Autobag: Learning to open plastic bags and insert objects,” in2023 IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 3918–3925

    2023

  70. [70]

    Blox-net: Generative design-for-robot-assembly using vlm supervision, physics simulation, and a robot with reset,

    A. Goldberg, K. Kondap, T. Qiu, Z. Ma, L. Fu, J. Kerr, H. Huang, K. Chen, K. Fang, and K. Goldberg, “Blox-net: Generative design-for-robot-assembly using vlm supervision, physics simulation, and a robot with reset,” in2025 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2025, pp. 15 493–15 500

    2025

  71. [71]

    Bomp: Bin-optimized motion planning,

    Z. Tam, K. Dharmarajan, T. Qiu, Y . Avigal, J. Ichnowski, and K. Goldberg, “Bomp: Bin-optimized motion planning,” in2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2024, pp. 11 056–11 063

    2024

  72. [72]

    Learning to efficiently plan robust frictional multi-object grasps,

    W. C. Agboh, S. Sharma, K. Srinivas, M. Parulekar, G. Datta, T. Qiu, J. Ichnowski, E. Solowjow, M. Dogar, and K. Goldberg, “Learning to efficiently plan robust frictional multi-object grasps,” in2023 IEEE/RSJ International Confer- ence on Intelligent Robots and Systems (IROS), IEEE, 2023, pp. 10 660–10 667

    2023

  73. [73]

    Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems,

    Q. Bu, J. Cai, L. Chen, X. Cui, Y . Ding, S. Feng, X. He, X. Huang, et al., “Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems,” in2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2025

    2025

  74. [74]

    Droid: A large-scale in-the-wild robot manipulation dataset,

    A. Sharma, V . Sundaresan, Y . Zhu, P. Shah, K. Liu, M. Laskin, J. Tompson, A. Wahid, Y . Chebotar, and K. Hausman, “Droid: A large-scale in-the-wild robot manipulation dataset,”arXiv preprint arXiv:2310.01894, 2023

    Pith/arXiv arXiv 2023

  75. [75]

    An open-source research kit for the da vinci® surgical system,

    P. Kazanzides, Z. Chen, A. Deguet, G. S. Fischer, R. H. Taylor, and S. P. DiMaio, “An open-source research kit for the da vinci® surgical system,” in2014 IEEE International Conference on Robotics and Automation (ICRA), 2014, pp. 6434–6439

    2014

  76. [76]

    Qwen2. 5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al., “Qwen2. 5-vl technical report,”arXiv preprint arXiv:2502.13923, 2025

    Pith/arXiv arXiv 2025

  77. [77]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al., “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  78. [78]

    Robobrain 2.5: Depth in sight, time in mind,

    H. Tan, E. Zhou, Z. Li, Y . Xu, Y . Ji, X. Chen, C. Chi, P. Wang, H. Jia, Y . Ao, et al., “Robobrain 2.5: Depth in sight, time in mind,”arXiv preprint arXiv:2601.14352, 2026

    arXiv 2026

  79. [79]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,

    G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al., “Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,”arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  80. [80]

    To think or not to think: A study of thinking in rule- based visual reinforcement fine-tuning,

    M. Li, J. Zhong, S. Zhao, Y . Lai, H. Zhang, W. B. Zhu, and K. Zhang, “To think or not to think: A study of thinking in rule- based visual reinforcement fine-tuning,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

Showing first 80 references.

This paper was first reviewed by grok-4.5 on July 11, 2026.