Pith. sign in

REVIEW 4 major objections 5 minor 137 references

AntiGrounding: Lifting Robotic Actions into VLM Representation Space for Decision Making

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read AntiGrounding lets a pretrained vision-language model pick robot trajectories directly from rendered images.

desk verdict A plausible system paper with an honest limitations section, but the core claim—VLM scores rank trajectories well enough to drive MPC—is never directly tested, and the reported numbers have inconsistencies. read the letter →

arxiv 2506.12374 v2 pith:ZRISXQ45 submitted 2025-06-14 cs.RO cs.AI

classification cs.ROcs.AI
keywords vision-languagemodelsroboticmanipulationtrajectoryevaluationvisualquestionansweringmodelpredictivecontrolzero-shotlearningreal2sim2realclosed-loop
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a pre-trained vision-language model (VLM) can act as the decision-maker for fine-grained robot manipulation without task-specific training or intermediate symbolic representations. The idea is to 'anti-ground': instead of compressing an instruction into skills or constraints, the system renders each candidate robot trajectory as colored overlays on images from several camera views and lets the VLM answer structured questions about safety, task alignment, efficiency, and physical feasibility. The scoring is wrapped in a closed-loop model predictive controller, and an offline meta-review refines the scoring template from logged experience. Across eight tabletop tasks, the method reports a 57.5% overall success rate, versus 41.25% for the strongest baseline family tested. If the claim holds, general-purpose VLMs can serve as trajectory evaluators for novel tasks with zero fine-tuning.

What carries the argument

The load-bearing object is the trajectory-based structured VQA evaluation inside the MPC loop. Candidate end-effector trajectories are sampled under an annealed exploration bias, filtered by collision and target-interaction constraints, and then color-coded and rendered onto multi-view scene images. A VLM ensemble answers four weighted sub-questions—safety, task alignment, efficiency, and physical feasibility—plus a viewpoint-clarity question; viewpoint confidence weights suppress unreliable views, and the aggregated score ranks trajectories, with the top scorer selected by the argmax rule. The 'anti-grounding' move is that the robot action itself becomes the visual input to the VLM rather than being converted to a compressed intermediate representation.

What would settle it

Take a fixed scene, generate candidate trajectories whose true success or failure is known from execution, render them with the paper's multi-view protocol, and ask the VLM ensemble to rank them. If the top-ranked trajectory fails more often than a randomly chosen feasible trajectory, or if ranking accuracy on fine-grained geometric perturbations is at chance, the central claim is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that reversing the instruction-grounding pipeline removes the information bottleneck. The paper argues that existing methods project a language instruction into compressed intermediate forms—symbolic skill sequences, voxel value maps, or keypoint constraints—and that this discards fine-grained spatial and physical detail the VLM has already encoded. AntiGrounding instead lifts candidate action trajectories into the VLM's native representation space by rendering them from robot-centric viewpoints and evaluating the augmented images with a structured visual-question-answering template. Scores from an ensemble of VLMs are aggregated across sub-questions, views weighted by viewpoint confidence, and agents, and the argmax trajectory is executed in a closed-loop MPC. The paper reports that this zero-shot procedure outperforms all three baseline families on the tested manipulation suite, and that the optional offline policy refinement further raises success rates by adjusting the evaluation template from past execution outcomes.

Load-bearing premise

The whole system rests on the VLM's rendered-image scores being good enough to rank trajectories that differ by only centimeter-level placement and degree-level orientation; if those scores do not track real physical feasibility, the argmax selection picks the wrong motion.

Editorial extensions

If this is right

  • A single pretrained VLM can serve as the cost function for closed-loop trajectory optimization, so a new manipulation task requires only a language instruction rather than reward engineering or fine-tuning.
  • Multi-view rendering with viewpoint confidence weighting lets the VLM infer 3D scene structure without a dedicated 3D encoder.
  • The offline meta-VLM refinement loop lets the evaluation policy self-correct after failures, so long-horizon performance improves with accumulated experience.
  • On the reported task suite, the method beats approaches built on symbolic skill sequences, voxel value maps, and keypoint constraints, with the largest gaps on tasks requiring precise alignment and multi-stage sequencing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The mechanism is not tied to a specific robot arm: any planner that can propose candidate trajectories and render them could adopt the same VLM-as-judge loop, potentially extending to mobile manipulation, navigation, or bimanual tasks.
  • A direct stress test would measure how the VLM ensemble's trajectory ranking degrades as candidate differences shrink below a few centimeters or degrees; the paper's own error analysis suggests this metric-level discrimination is the fragile point.
  • The offline refinement could become an online self-supervised loop that uses execution success flags as reward labels to tune template weights, a step the paper leaves for future work.
  • Because evaluation is purely image-and-text based, the framework could combine with other search strategies beyond the spherical sampler to explore higher-dimensional or contact-rich trajectory spaces.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AntiGrounding, a framework that inverts conventional instruction-grounding pipelines by rendering candidate robot trajectories from multiple viewpoints and asking a pre-trained VLM ensemble to score them via structured visual question answering. The highest-scoring trajectory is executed in a closed-loop MPC, with an optional offline meta-VLM module that refines the evaluation template from prior execution logs. The authors report success rates on eight real-robot manipulation tasks, comparing against Code as Policies, Voxposer, and Rekep, and claim zero-shot synthesis of closed-loop trajectories for novel tasks. The paper also includes ablations, an offline policy-optimization study, and a system error breakdown.

Significance. If the central premise holds, the paper contributes an interesting alternative to intermediate-representation approaches: instead of compressing VLM knowledge into symbolic skills or value maps, it uses rendered trajectory overlays as a native interface to the VLM's spatial and semantic priors. The use of an ensemble of five pre-trained VLMs in a closed-loop MPC, without task-specific fine-tuning, is a distinctive and potentially useful design point. The paper also honestly documents its own limitations in Section 4.4 and Appendix A.8, including that VLM evaluation errors dominate and that fine-grained metric geometric constraints remain a challenge. However, the empirical support for the central premise is currently incomplete: the manuscript never directly measures whether the VLM scores in Eq. (3) are positively correlated with physical task success, and the reported numerical comparisons suffer from inconsistencies and lack of statistical rigor. The concept is promising, but the evidence as presented is not yet commensurate with the strength of the central claim.

major comments (4)
  1. [Section 3.3, Eqs. (2)-(4); Section 4.4; Appendix A.8] The load-bearing assumption is that the VLM scores S_{j,t} in Eq. (3) and the argmax in Eq. (4) reliably rank physically feasible trajectories. The paper never validates this correlation directly, instead reporting end-of-task success that couples VLM scoring with trajectory sampling, constraint filtering, and execution. Section 4.4 states that VLM evaluation errors dominate the failure breakdown, and Appendix A.8 concedes that "fine-grained, metric geometric constraints (e.g., sub-centimeter concentricity...) purely from 2D renderings remains a significant challenge for current VLMs." This is precisely the regime the method targets. I ask the authors to provide a direct scoring-accuracy measurement, for example by comparing VLM rankings against ground-truth or human-annotated trajectory quality on a held-out set of candidate trajectories, with per-task correlation coefficients or top-k selection accuracy. Without this, the central claim that anti-grounding enables reliable decision making remains asserted rather than demonstrated.
  2. [Table 1 vs. Table 3] The success numbers are internally inconsistent. Table 1 reports AntiGrounding success of 6/10 for Circular Tape Stacking, while Table 3 reports 55.24% for the "No Prior" condition on the same task; Waste Sorting is 7/10 in Table 1 but 68.49% in Table 3. Additionally, in Table 1, Rekep beats AntiGrounding on Water Pouring (6/10 vs. 5/10) despite AntiGrounding's higher aggregate score. The authors need to clarify whether these tables use different trial counts, different task configurations, or different evaluation criteria, and report the exact number of trials and the definition of success for each table. As written, the inconsistencies undermine the quantitative comparison.
  3. [Section 4.1, Table 1; Section 4.2, Table 2] The main comparison in Table 1 is based on 10 trials per task with no confidence intervals, error bars, or significance tests, and Table 2 reports ablation variants from a single run while only AntiGrounding is averaged over three runs. Given the high variance suggested by the AntiGrounding ablation standard deviations in Table 2 (e.g., 46.67±9.87 for Water Pouring), the differences between AntiGrounding and Rekep on individual tasks (e.g., 5/10 vs. 6/10 on Water Pouring) could easily be noise. The authors should provide per-task confidence intervals and a significance test (or at least additional repetitions) for the main comparison, and report multiple runs for the ablations.
  4. [Section 4.3 and Appendix A.6] The offline policy refinement is evaluated on the same tasks whose execution outcomes are used to tune the sub-question weights and templates, which risks overfitting and makes the "improvement" numbers in Table 3 partly circular. The paper should clarify whether the offline refinement was evaluated on held-out task instances or only on the same task set, and ideally report performance on tasks not used for refinement. In addition, Table 3 does not state the number of trials per entry or whether the same trial set was used for both conditions; these details are necessary to assess the reported improvements.
minor comments (5)
  1. [Section 1] The sentence "Extensive experiments in both simulation and real-world environments demonstrate that AntiGrounding significantly outperforms conventional methods... particularly in complex manipulation tasks in complex manipulation tasks encompassing precise spatial reasoning" contains a duplicated phrase that should be removed.
  2. [Figure 3 caption] The caption contains a typo: "Sim Buider" should be "Sim Builder."
  3. [Appendix A.5.1] The sentence "Each model processes the same pipline in parallel" contains a typo: "pipline" should be "pipeline."
  4. [Appendix A.2.3] The Water Pouring task is described as pouring from a "green cup" into a "stainless steel bowl," whereas Section 1 and Figure 7 refer to a teapot and cup; please reconcile the object descriptions across the paper.
  5. [Section 4.2] The ablation paragraph says AG-MVECW "led to unstable trajectory assessments" and AG-VGTG "severely affected trajectory quality," but the corresponding rows in Table 2 do not include error bars or significance tests; the text should be moderated to match the evidence presented.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central zero-shot claim is externally evaluated, and the optional refinement module's in-domain tuning is a limitation, not a circular derivation.

full rationale

The paper's main derivation is an MPC loop in which pre-trained VLMs score externally rendered candidate trajectories (Eqs. 2-4), and success is measured by external task completion, not by the VLM's own scores. No parameter of the main method is fitted to the success labels that are then reported as the headline result: the VLM ensemble and hand-set template weights are fixed for the zero-shot comparisons in Table 1, and the baselines are independent systems. There is no load-bearing self-citation chain: the cited prior work on VLMs, spatial reasoning, Real2Sim, and MPC is external to this paper's authors. The strongest caveats are empirical, not circular: Section 4.4 reports that VLM evaluation errors dominate failures, and Appendix A.8 concedes that fine-grained metric geometric evaluation from 2D renderings remains a significant VLM challenge. Those concessions weaken the premise that VLM scores can rank centimeter-level trajectories, but they do not make the derivation reduce to its inputs. The optional offline policy refinement (Sec. 4.3, Appendix A.6) updates template weights from execution outcomes on the same task set and then reports improved success on those tasks; this is an in-domain fitting/adaptation evaluation rather than a held-out prediction, and it is not presented as the zero-shot claim. It is a methodological limitation, not a circular step under the strict definition used here.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The framework's decision loop rests on domain assumptions about VLM spatial judgment, simulator fidelity, and rigid grasping, plus hand-set sampling and weighting parameters that affect the score aggregation in Eq. (3). No new physical entities are introduced.

free parameters (5)
  • Sub-question weights w_k = safety 0.25, task alignment 0.35, efficiency 0.20, physical feasibility 0.20 (initial, later tuned by meta-VLM)
    These weights directly determine the aggregate trajectory score in Eq. (3); they are hand-set initially and adjusted in offline refinement based on observed failures on the evaluation tasks themselves.
  • Sampling radius and angle schedule = R0=0.25 m, Rmin=0.1 m, lambda_R=0.693, theta0=90 deg, theta_min=30 deg, lambda_theta=0.712
    Exploration hyperparameters for trajectory candidate generation (Appendix A.4.1); the quality of the candidate set depends on them.
  • Viewpoint confidence decay lambda_C = 0.5
    Controls how strongly inter-agent variance reduces viewpoint confidence in Eq. (2) (Appendix A.5.2).
  • Number of active views k = 3
    Adaptive viewpoint selection budget (Algorithm 1), a hand-chosen evaluation budget.
  • Prediction horizon H_t and candidate count N = unspecified
    Sampling parameters for trajectory candidates are mentioned in Section 3.2 but not given values; the candidate set size influences whether the argmax in Eq. (4) has good options.
assumptions (5)
  • domain assumption VLM ensemble members return parseable, consistent numeric scores for the structured VQA template.
    The whole score aggregation assumes each of the five proprietary VLMs emits scores from 1 to 10 for every sub-question and view; the prompt template (Listing 3) instructs them to, but this is an unverified property of external APIs.
  • domain assumption Rendering candidate trajectories as colored overlays on multi-view images preserves enough spatial and geometric information for the VLM to judge them.
    Central to Section 3.3; this is a premise about VLM perception and rendering fidelity.
  • domain assumption The Real2Sim2Real pipeline (SPAR3D reconstruction, Scalable Real2Sim physical identification, SAM-6D pose alignment) is accurate enough that simulated evaluation transfers to the real robot.
    Used in Section 3.1; the paper's own limitations list this as a key source of error.
  • domain assumption Rigid coupling between the grasped object and the end-effector is maintained during manipulation.
    Stated in Section 3.4; if slippage occurs, the rigid coupling rotation module produces wrong orientations, as the paper acknowledges for deformable objects.
  • ad hoc to paper Prompt rule requiring significant score variance across trajectories does not distort rankings.
    The template in Appendix A.5.3 (Rule 6) forces VLMs to avoid uniform scores; this is a hand-added constraint with no evidence that it improves ranking fidelity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AntiGrounding: Lifting Robotic Actions into VLM Representation Space for Decision Making." pith.science (2026). https://pith.science/paper/ZRISXQ45

@misc{pith2026250612374,
  author       = {Pith},
  title        = {Pith review of: AntiGrounding: Lifting Robotic Actions into VLM Representation Space for Decision Making},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZRISXQ45}},
  note         = {Machine review of arXiv:2506.12374}
}
read the original abstract

Vision-Language Models (VLMs) encode knowledge and reasoning capabilities for robotic manipulation within high-dimensional representation spaces. However, current approaches often project them into compressed intermediate representations, discarding important task-specific information such as fine-grained spatial or semantic details. To address this, we propose AntiGrounding, a new framework that reverses the instruction grounding process. It lifts candidate actions directly into the VLM representation space, renders trajectories from multiple views, and uses structured visual question answering for instruction-based decision making. This enables zero-shot synthesis of optimal closed-loop robot trajectories for new tasks. We also propose an offline policy refinement module that leverages past experience to enhance long-term performance. Experiments in both simulation and real-world environments show that our method outperforms baselines across diverse robotic manipulation tasks.

Figures

Figures reproduced from arXiv: 2506.12374 by the authors.

Figure 1
Figure 1. AntiGrounding first builds the Real2Sim2Real pipeline, which includes 3D reconstruction, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The diagram illustrates the intrinsic capabilities of VLM (more discussion in Appendix [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of AntiGrounding. The framework follows a real2sim2real pipeline, integrating real-world perception, precise simulation, and adaptive offline policy refinement. It includes geo￾metric reconstruction, physical property identification, and 6D pose alignment for scene building. Trajectories are generated through multi-stage generation, evaluated using structured VQA templates, and executed under a closed-loop … view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The diagram illustrates the rigid coupling rotation module. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Among these, VLM errors dominate, reflecting challenges in trajectory assessment, multi [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Real-World and Simulated Environment Setup [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Visualization of action execution across four representative tasks: Circular Tape Stacking, [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Parameter Diagram A.4.2 Constraint Environment Collision (Ccoll): Checking for collisions between the robot body and non-target obstacles represented in the dynamic scene. For instance, this can be expressed as: Ccoll(T n t ) = [ p∈T n t  1, if distance(p, Et) < ϵ 0, …
Figure 9
Figure 9. Figure 9: Multi-view rendering of candidate trajectories at a single time step. [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: View-wise scoring heatmaps returned by the VLM for each candidate trajectory under [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Comparison between an imprecise reconstructed mesh (left) and the accurate reconstructed [PITH_FULL_IMAGE:figures/full_fig_p030_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

137 extracted references · 9 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning.Advances in Neural Information Processing Systems, 35:23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in Neural Information Processing Systems, 35:23716–23736, 2022

  2. [2]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning, pages 8748–8763, 2021

  3. [3]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. InInternational Conference on Machine Learning, pages 4904–4916, 2021

  4. [4]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. InInternational Conference on Machine Learning, pages 8821–8831. PMLR, 2021

  5. [5]

    Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023

  6. [6]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

  7. [7]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

  8. [8]

    Do as i can, not as i say: Grounding language in robotic affordances.arXiv preprint arXiv:2204.01691, 2022

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances.arXiv preprint arXiv:2204.01691, 2022

Show all 137 references
  1. [9]

    Open-vocabulary queryable scene representations for real world planning

    Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao, Keerthana Gopalakrishnan, Michael S Ryoo, Austin Stone, and Daniel Kappler. Open-vocabulary queryable scene representations for real world planning. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages ...

  2. [10]

    Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action

    Dhruv Shah, Bła ˙zej Osi´nski, Sergey Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. InConference on robot learning, pages 492–504. PMLR, 2023

  3. [11]

    So- cratic models: Composing zero-shot multimodal reasoning with language.arXiv preprint arXiv:2204.00598, 2022

    Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. So- cratic models: Composing zero-shot multimodal reasoning with language.arXiv preprint arXiv:2204.00598, 2022

  4. [12]

    Language models as zero-shot planners: Extracting actionable knowledge for embodied agents

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. InInternational conference on machine learning, pages 9118–9147. PMLR, 2022

  5. [13]

    Code as policies: Language model programs for embodied control

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493–9500. IEEE, 2023. 10

  6. [14]

    V oxposer: Composable 3d value maps for robotic manipulation with language models.arXiv preprint arXiv:2307.05973, 2023

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models.arXiv preprint arXiv:2307.05973, 2023

  7. [15]

    Pivot: Iterative visual prompting elicits actionable knowledge for vlms.arXiv preprint arXiv:2402.07872, 2024

    Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms.arXiv preprint arXiv:2402.07872, 2024

  8. [16]

    Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation.arXiv preprint arXiv:2409.01652, 2024

    Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation.arXiv preprint arXiv:2409.01652, 2024

  9. [17]

    Robodexvlm: Visual language model-enabled task planning and motion control for dexterous robot manipu- lation.arXiv preprint arXiv:2503.01616, 2025

    Haichao Liu, Sikai Guo, Pengfei Mai, Jiahang Cao, Haoang Li, and Jun Ma. Robodexvlm: Visual language model-enabled task planning and motion control for dexterous robot manipu- lation.arXiv preprint arXiv:2503.01616, 2025

  10. [18]

    Roboground: Robotic manipulation with grounded vision- language priors.arXiv preprint arXiv:2504.21530, 2025

    Haifeng Huang, Xinyi Chen, Yilun Chen, Hao Li, Xiaoshen Han, Zehan Wang, Tai Wang, Jiangmiao Pang, and Zhou Zhao. Roboground: Robotic manipulation with grounded vision- language priors.arXiv preprint arXiv:2504.21530, 2025

  11. [19]

    Llm-grounder: Open-vocabulary 3d visual grounding with large lan- guage model as an agent

    Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F Fouhey, and Joyce Chai. Llm-grounder: Open-vocabulary 3d visual grounding with large lan- guage model as an agent. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 7...

  12. [20]

    Spatialpin: Enhancing spatial reasoning capabilities of vision-language models through prompting and interacting 3d priors.arXiv preprint arXiv:2403.13438, 2024

    Chenyang Ma, Kai Lu, Ta-Ying Cheng, Niki Trigoni, and Andrew Markham. Spatialpin: Enhancing spatial reasoning capabilities of vision-language models through prompting and interacting 3d priors.arXiv preprint arXiv:2403.13438, 2024

  13. [21]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455–14465, 2024

  14. [22]

    Spatialrgpt: Grounded spatial reasoning in vision language models.arXiv preprint arXiv:2406.01584, 2024

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language models.arXiv preprint arXiv:2406.01584, 2024

  15. [23]

    Capture: Evaluating spatial reasoning in vision language models via occluded object counting.arXiv preprint arXiv:2504.15485, 2025

    Atin Pothiraj, Elias Stengel-Eskin, Jaemin Cho, and Mohit Bansal. Capture: Evaluating spatial reasoning in vision language models via occluded object counting.arXiv preprint arXiv:2504.15485, 2025

  16. [24]

    Mind the gap: Benchmarking spatial reasoning in vision-language models.arXiv preprint arXiv:2503.19707, 2025

    Ilias Stogiannidis, Steven McDonagh, and Sotirios A Tsaftaris. Mind the gap: Benchmarking spatial reasoning in vision-language models.arXiv preprint arXiv:2503.19707, 2025

  17. [25]

    Zero-shot visual reasoning by vision- language models: Benchmarking and analysis

    Aishik Nagar, Shantanu Jaiswal, and Cheston Tan. Zero-shot visual reasoning by vision- language models: Benchmarking and analysis. In2024 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2024

  18. [26]

    How to enable llm with 3d capacity? a survey of spatial reasoning in llm.arXiv preprint arXiv:2504.05786, 2025

    Jirong Zha, Yuxuan Fan, Xiao Yang, Chen Gao, and Xinlei Chen. How to enable llm with 3d capacity? a survey of spatial reasoning in llm.arXiv preprint arXiv:2504.05786, 2025

  19. [27]

    Thinking in space: How multimodal large language models see, remember, and recall spaces.arXiv preprint arXiv:2412.14171, 2024

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces.arXiv preprint arXiv:2412.14171, 2024

  20. [28]

    Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024

    Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024

  21. [29]

    Agent3d-zero: An agent for zero-shot 3d understanding

    Sha Zhang, Di Huang, Jiajun Deng, Shixiang Tang, Wanli Ouyang, Tong He, and Yanyong Zhang. Agent3d-zero: An agent for zero-shot 3d understanding. InEuropean Conference on Computer Vision, pages 186–202. Springer, 2024

  22. [30]

    Shapellm: Universal 3d object understanding for embodied interaction

    Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. Shapellm: Universal 3d object understanding for embodied interaction. In European Conference on Computer Vision, pages 214–238. Springer, 2024

  23. [31]

    Scene-llm: Extending language model for 3d visual understanding and reasoning.arXiv preprint arXiv:2403.11401, 2024

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning.arXiv preprint arXiv:2403.11401, 2024. 11

  24. [32]

    Inf-llava: Dual-perspective perception for high-resolution multimodal large language model

    Yiwei Ma, Zhibin Wang, Xiaoshuai Sun, Weihuang Lin, Qiang Zhou, Jiayi Ji, and Rongrong Ji. Inf-llava: Dual-perspective perception for high-resolution multimodal large language model. arXiv preprint arXiv:2407.16198, 2024

  25. [33]

    G-llava: Solving geometric problem with multi-modal large language model.arXiv preprint arXiv:2312.11370, 2023

    Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al. G-llava: Solving geometric problem with multi-modal large language model.arXiv preprint arXiv:2312.11370, 2023

  26. [34]

    Manipvqa: Injecting robotic affordance and physically grounded information into multi-modal large language models

    Siyuan Huang, Iaroslav Ponomarenko, Zhengkai Jiang, Xiaoqi Li, Xiaobin Hu, Peng Gao, Hongsheng Li, and Hao Dong. Manipvqa: Injecting robotic affordance and physically grounded information into multi-modal large language models. In2024 IEEE/RSJ International Conference on Intel...

  27. [35]

    Robovqa: Multimodal long-horizon reasoning for robotics

    Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakr- ishnan, Christine Chan, Gabriel Dulac-Arnold, Sharath Maddineni, Nikhil J Joshi, et al. Robovqa: Multimodal long-horizon reasoning for robotics. In2024 IEEE International Conference on ...

  28. [36]

    Mqa: Answering the question via robotic manipulation.arXiv preprint arXiv:2003.04641, 2020

    Yuhong Deng, Di Guo, Xiaofeng Guo, Naifu Zhang, Huaping Liu, and Fuchun Sun. Mqa: Answering the question via robotic manipulation.arXiv preprint arXiv:2003.04641, 2020

  29. [37]

    Robotvqa—a scene-graph-and deep-learning-based visual question answering system for robot manipulation

    Franklin Kenghagho Kenfack, Feroz Ahmed Siddiky, Ferenc Balint-Benczedi, and Michael Beetz. Robotvqa—a scene-graph-and deep-learning-based visual question answering system for robot manipulation. In2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)...

  30. [38]

    A visual questioning answering approach to enhance robot localization in indoor environments.Frontiers in Neurorobotics, 17:1290584, 2023

    Juan Diego Peña-Narvaez, Francisco Martín, José Miguel Guerrero, and Rodrigo Pérez- Rodríguez. A visual questioning answering approach to enhance robot localization in indoor environments.Frontiers in Neurorobotics, 17:1290584, 2023

  31. [39]

    Vlmpc: Vision-language model predictive control for robotic manipulation.arXiv preprint arXiv:2407.09829, 2024

    Wentao Zhao, Jiaming Chen, Ziyu Meng, Donghui Mao, Ran Song, and Wei Zhang. Vlmpc: Vision-language model predictive control for robotic manipulation.arXiv preprint arXiv:2407.09829, 2024

  32. [40]

    Re- flective planning: Vision-language models for multi-stage long-horizon robotic manipulation

    Yunhai Feng, Jiaming Han, Zhuoran Yang, Xiangyu Yue, Sergey Levine, and Jianlan Luo. Re- flective planning: Vision-language models for multi-stage long-horizon robotic manipulation. arXiv preprint arXiv:2502.16707, 2025

  33. [41]

    Open-world task and motion planning via vision-language model inferred constraints.arXiv preprint arXiv:2411.08253, 2024

    Nishanth Kumar, William Shen, Fabio Ramos, Dieter Fox, Tomás Lozano-Pérez, Leslie Pack Kaelbling, and Caelan Reed Garrett. Open-world task and motion planning via vision-language model inferred constraints.arXiv preprint arXiv:2411.08253, 2024

  34. [42]

    Rt-2: Vision-language- action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language- action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023

  35. [43]

    Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

  36. [44]

    Robomamba: Multimodal state space model for efficient robot reasoning and manipulation.arXiv preprint arXiv:2406.04339, 2024

    Jiaming Liu, Mengzhen Liu, Zhenyu Wang, Lily Lee, Kaichen Zhou, Pengju An, Senqiao Yang, Renrui Zhang, Yandong Guo, and Shanghang Zhang. Robomamba: Multimodal state space model for efficient robot reasoning and manipulation.arXiv preprint arXiv:2406.04339, 2024

  37. [45]

    Scaling proprioceptive-visual learn- ing with heterogeneous pre-trained transformers.Advances in Neural Information Processing Systems, 37:124420–124450, 2024

    Lirui Wang, Xinlei Chen, Jialiang Zhao, and Kaiming He. Scaling proprioceptive-visual learn- ing with heterogeneous pre-trained transformers.Advances in Neural Information Processing Systems, 37:124420–124450, 2024

  38. [46]

    Learning fine-grained bimanual manipulation with low-cost hardware.arXiv preprint arXiv:2304.13705, 2023

    Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware.arXiv preprint arXiv:2304.13705, 2023

  39. [47]

    Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation.arXiv preprint arXiv:2401.02117, 2024

    Zipeng Fu, Tony Z Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation.arXiv preprint arXiv:2401.02117, 2024

  40. [48]

    Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024. 12

  41. [49]

    Rdt-1b: a diffusion foundation model for bimanual manipulation

    Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864, 2024

  42. [50]

    Diffusion-vla: Scaling robot foundation models via unified diffusion and autoregression.arXiv preprint arXiv:2412.03293, 2024

    Junjie Wen, Minjie Zhu, Yichen Zhu, Zhibin Tang, Jinming Li, Zhongyi Zhou, Chengmeng Li, Xiaoyu Liu, Yaxin Peng, Chaomin Shen, et al. Diffusion-vla: Scaling robot foundation models via unified diffusion and autoregression.arXiv preprint arXiv:2412.03293, 2024

  43. [51]

    π0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

  44. [52]

    π0.5: A vision- language-action model with open-world generalization.arXiv preprint arXiv:2504.16054, 2025

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. π0.5: A vision- language-action model with open-world generalization.arXiv preprint arXiv:2504.16054, 2025

  45. [53]

    Manipulate-anything: Automating real-world robots using vision-language models.arXiv preprint arXiv:2406.18915, 2024

    Jiafei Duan, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, and Ranjay Krishna. Manipulate-anything: Automating real-world robots using vision-language models.arXiv preprint arXiv:2406.18915, 2024

  46. [54]

    Skillman—a skill-based robotic manipulation framework based on perception and reasoning.Robotics and Autonomous Systems, 134:103653, 2020

    Mohammed Diab, Mihai Pomarlan, Daniel Beßler, Aliakbar Akbari, Jan Rosell, John Bate- man, and Michael Beetz. Skillman—a skill-based robotic manipulation framework based on perception and reasoning.Robotics and Autonomous Systems, 134:103653, 2020

  47. [55]

    A real-to-sim-to-real approach to robotic manipulation with vlm-generated iterative keypoint rewards.arXiv preprint arXiv:2502.08643, 2025

    Shivansh Patel, Xinchen Yin, Wenlong Huang, Shubham Garg, Hooshang Nayyeri, Li Fei-Fei, Svetlana Lazebnik, and Yunzhu Li. A real-to-sim-to-real approach to robotic manipulation with vlm-generated iterative keypoint rewards.arXiv preprint arXiv:2502.08643, 2025

  48. [56]

    Sofar: Language-grounded orientation bridges spatial reasoning and object manipulation.arXiv preprint arXiv:2502.13143, 2025

    Zekun Qi, Wenyao Zhang, Yufei Ding, Runpei Dong, Xinqiang Yu, Jingwen Li, Lingyun Xu, Baoyu Li, Xialin He, Guofan Fan, et al. Sofar: Language-grounded orientation bridges spatial reasoning and object manipulation.arXiv preprint arXiv:2502.13143, 2025

  49. [57]

    Geomanip: Geometric constraints as general interfaces for robot manipulation

    Weiliang Tang, Jia-Hui Pan, Yun-Hui Liu, Masayoshi Tomizuka, Li Erran Li, Chi-Wing Fu, and Mingyu Ding. Geomanip: Geometric constraints as general interfaces for robot manipulation. arXiv preprint arXiv:2501.09783, 2025

  50. [58]

    Kuda: Keypoints to unify dynamics learning and visual prompting for open-vocabulary robotic manipulation.arXiv preprint arXiv:2503.10546, 2025

    Zixian Liu, Mingtong Zhang, and Yunzhu Li. Kuda: Keypoints to unify dynamics learning and visual prompting for open-vocabulary robotic manipulation.arXiv preprint arXiv:2503.10546, 2025

  51. [59]

    Robogsim: A real2sim2real robotic gaussian splatting simulator

    Xinhai Li, Jialin Li, Ziheng Zhang, Rui Zhang, Fan Jia, Tiancai Wang, Haoqiang Fan, Kuo-Kun Tseng, and Ruiping Wang. Robogsim: A real2sim2real robotic gaussian splatting simulator. arXiv preprint arXiv:2411.11839, 2024

  52. [60]

    Rl-gsbridge: 3d gaussian splatting based real2sim2real method for robotic manipulation learning.arXiv preprint arXiv:2409.20291, 2024

    Yuxuan Wu, Lei Pan, Wenhua Wu, Guangming Wang, Yanzi Miao, Fan Xu, and Hesheng Wang. Rl-gsbridge: 3d gaussian splatting based real2sim2real method for robotic manipulation learning.arXiv preprint arXiv:2409.20291, 2024

  53. [61]

    Discovery and deployment of emergent robot swarm behaviors via represen- tation learning and real2sim2real transfer.arXiv preprint arXiv:2502.15937, 2025

    Connor Mattson, Varun Raveendra, Ricardo Vega, Cameron Nowzari, Daniel S Drew, and Daniel S Brown. Discovery and deployment of emergent robot swarm behaviors via represen- tation learning and real2sim2real transfer.arXiv preprint arXiv:2502.15937, 2025

  54. [62]

    Reconciling reality through simulation: A real-to-sim-to-real approach for robust manipulation.arXiv preprint arXiv:2403.03949, 2024

    Marcel Torne, Anthony Simeonov, Zechu Li, April Chan, Tao Chen, Abhishek Gupta, and Pulkit Agrawal. Reconciling reality through simulation: A real-to-sim-to-real approach for robust manipulation.arXiv preprint arXiv:2403.03949, 2024

  55. [63]

    Rl-vigen: A reinforcement learning benchmark for visual generalization.Advances in Neural Information Processing Systems, 36:6720–6747, 2023

    Zhecheng Yuan, Sizhe Yang, Pu Hua, Can Chang, Kaizhe Hu, and Huazhe Xu. Rl-vigen: A reinforcement learning benchmark for visual generalization.Advances in Neural Information Processing Systems, 36:6720–6747, 2023

  56. [64]

    Isaac gym: High performance gpu-based physics simulation for robot learning.arXiv preprint arXiv:2108.10470, 2021

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning.arXiv preprint arXiv:2108.10470, 2021

  57. [65]

    Efficient real2sim2real of continuum robots using deep reinforcement learning with koopman operator.IEEE Transactions on Industrial Electronics, 2025

    Guanglin Ji, Qian Gao, Yin Xiao, and Zhenglong Sun. Efficient real2sim2real of continuum robots using deep reinforcement learning with koopman operator.IEEE Transactions on Industrial Electronics, 2025. 13

  58. [66]

    Real-time per- ception meets reactive motion generation.IEEE Robotics and Automation Letters, 3(3):1864– 1871, 2018

    Daniel Kappler, Franziska Meier, Jan Issac, Jim Mainprice, Cristina Garcia Cifuentes, Manuel Wüthrich, Vincent Berenz, Stefan Schaal, Nathan Ratliff, and Jeannette Bohg. Real-time per- ception meets reactive motion generation.IEEE Robotics and Automation Letters, 3(3):1864– 1871, 2018

  59. [67]

    You only demonstrate once: Category-level manipulation from single visual demonstration.arXiv preprint arXiv:2201.12716, 2022

    Bowen Wen, Wenzhao Lian, Kostas Bekris, and Stefan Schaal. You only demonstrate once: Category-level manipulation from single visual demonstration.arXiv preprint arXiv:2201.12716, 2022

  60. [68]

    One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion

    Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. InProceedings of the IEEE/CVF conference on compu...

  61. [69]

    Sparp: Fast 3d object reconstruction and pose estimation from sparse views

    Chao Xu, Ang Li, Linghao Chen, Yulin Liu, Ruoxi Shi, Hao Su, and Minghua Liu. Sparp: Fast 3d object reconstruction and pose estimation from sparse views. InEuropean Conference on Computer Vision, pages 143–163. Springer, 2024

  62. [70]

    Zero123++: a single image to consistent multi-view diffusion base model.arXiv preprint arXiv:2310.15110, 2023

    Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view diffusion base model.arXiv preprint arXiv:2310.15110, 2023

  63. [71]

    Zero-1-to-3: Zero-shot one image to 3d object

    Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl V ondrick. Zero-1-to-3: Zero-shot one image to 3d object. InProceedings of the IEEE/CVF international conference on computer vision, pages 9298–9309, 2023

  64. [72]

    Get3d: A generative model of high quality 3d textured shapes learned from images.Advances In Neural Information Processing Systems, 35:31841–31854, 2022

    Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images.Advances In Neural Information Processing Systems, 35:31841–31854, 2022

  65. [73]

    A-sdf: Learning disentangled signed distance functions for articulated shape represen- tation

    Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, and Xiaolong Wang. A-sdf: Learning disentangled signed distance functions for articulated shape represen- tation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 13001–13011, 2021

  66. [74]

    Ditto: Building digital twins of articulated objects from interaction

    Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5616–5626, 2022

  67. [75]

    Structure from action: Learn- ing interactions for articulated object 3d structure discovery.arXiv preprint arXiv:2207.08997, 2022

    Neil Nie, Samir Yitzhak Gadre, Kiana Ehsani, and Shuran Song. Structure from action: Learn- ing interactions for articulated object 3d structure discovery.arXiv preprint arXiv:2207.08997, 2022

  68. [76]

    Urdformer: A pipeline for constructing articulated simulation environments from real-world images.arXiv preprint arXiv:2405.11656, 2024

    Zoey Chen, Aaron Walsman, Marius Memmel, Kaichun Mo, Alex Fang, Karthikeya Vemuri, Alan Wu, Dieter Fox, and Abhishek Gupta. Urdformer: A pipeline for constructing articulated simulation environments from real-world images.arXiv preprint arXiv:2405.11656, 2024

  69. [77]

    Paris: Part-level reconstruction and motion analysis for articulated objects

    Jiayi Liu, Ali Mahdavi-Amiri, and Manolis Savva. Paris: Part-level reconstruction and motion analysis for articulated objects. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 352–363, 2023

  70. [78]

    Cage: controllable articulation generation

    Jiayi Liu, Hou In Ivan Tam, Ali Mahdavi-Amiri, and Manolis Savva. Cage: controllable articulation generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17880–17889, 2024

  71. [79]

    Sam-6d: Segment anything model meets zero-shot 6d object pose estimation

    Jiehong Lin, Lihua Liu, Dekun Lu, and Kui Jia. Sam-6d: Segment anything model meets zero-shot 6d object pose estimation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27906–27916, 2024

  72. [80]

    Gigapose: Fast and robust novel object pose estimation via one correspondence

    Van Nguyen Nguyen, Thibault Groueix, Mathieu Salzmann, and Vincent Lepetit. Gigapose: Fast and robust novel object pose estimation via one correspondence. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9903–9913, 2024

  73. [81]

    Any6d: Model-free 6d pose estimation of novel objects.arXiv preprint arXiv:2503.18673, 2025

    Taeyeop Lee, Bowen Wen, Minjun Kang, Gyuree Kang, In So Kweon, and Kuk-Jin Yoon. Any6d: Model-free 6d pose estimation of novel objects.arXiv preprint arXiv:2503.18673, 2025. 14

  74. [82]

    Foundationpose: Unified 6d pose estimation and tracking of novel objects

    Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17868–17879, 2024

  75. [83]

    Foundpose: Unseen object pose estimation with foundation features

    Evin Pınar Örnek, Yann Labbé, Bugra Tekin, Lingni Ma, Cem Keskin, Christian Forster, and Tomas Hodan. Foundpose: Unseen object pose estimation with foundation features. In European Conference on Computer Vision, pages 163–182. Springer, 2024

  76. [84]

    A comprehensive survey on point cloud registration.arXiv preprint arXiv:2103.02690, 2021

    Xiaoshui Huang, Guofeng Mei, Jian Zhang, and Rana Abbas. A comprehensive survey on point cloud registration.arXiv preprint arXiv:2103.02690, 2021

  77. [85]

    Deep learning-based point cloud registration: A comprehensive survey and taxonomy

    Yu-Xin Zhang, Jie Gui, Baosheng Yu, Xiaofeng Cong, Xin Gong, Wenbing Tao, and Dacheng Tao. Deep learning-based point cloud registration: A comprehensive survey and taxonomy. arXiv preprint arXiv:2404.13830, 2024

  78. [86]

    A tutorial review on point cloud registrations: principle, classification, comparison, and technology challenges.Mathematical Problems in Engineering, 2021(1):9953910, 2021

    Leihui Li, Riwei Wang, and Xuping Zhang. A tutorial review on point cloud registrations: principle, classification, comparison, and technology challenges.Mathematical Problems in Engineering, 2021(1):9953910, 2021

  79. [87]

    A comprehensive survey of visual slam algorithms.Robotics, 11(1):24, 2022

    Andréa Macario Barros, Maugan Michel, Yoann Moline, Gwenolé Corre, and Frédérick Carrel. A comprehensive survey of visual slam algorithms.Robotics, 11(1):24, 2022

  80. [88]

    How nerfs and 3d gaussian splatting are reshaping slam: a survey.arXiv preprint arXiv:2402.13255, 4:1, 2024

    Fabio Tosi, Youmin Zhang, Ziren Gong, Erik Sandström, Stefano Mattoccia, Martin R Oswald, and Matteo Poggi. How nerfs and 3d gaussian splatting are reshaping slam: a survey.arXiv preprint arXiv:2402.13255, 4:1, 2024

  81. [89]

    A survey on active simultaneous localization and mapping: State of the art and new frontiers.IEEE Transactions on Robotics, 39(3):1686–1705, 2023

    Julio A Placed, Jared Strader, Henry Carrillo, Nikolay Atanasov, Vadim Indelman, Luca Carlone, and José A Castellanos. A survey on active simultaneous localization and mapping: State of the art and new frontiers.IEEE Transactions on Robotics, 39(3):1686–1705, 2023

  82. [90]

    Scalable real2sim: Physics-aware asset generation via robotic pick-and-place setups.arXiv preprint arXiv:2503.00370, 2025

    Nicholas Pfaff, Evelyn Fu, Jeremy Binagia, Phillip Isola, and Russ Tedrake. Scalable real2sim: Physics-aware asset generation via robotic pick-and-place setups.arXiv preprint arXiv:2503.00370, 2025

  83. [91]

    Phystwin: Physics-informed reconstruction and simulation of deformable objects from videos

    Hanxiao Jiang, Hao-Yu Hsu, Kaifeng Zhang, Hsin-Ni Yu, Shenlong Wang, and Yunzhu Li. Phystwin: Physics-informed reconstruction and simulation of deformable objects from videos. arXiv preprint arXiv:2503.17973, 2025

  84. [92]

    Sim2real 2: Actively building explicit physics model for precise articulated object manipulation

    Liqian Ma, Jiaojiao Meng, Shuntao Liu, Weihang Chen, Jing Xu, and Rui Chen. Sim2real 2: Actively building explicit physics model for precise articulated object manipulation. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11698–11704. IEEE, 2023

  85. [93]

    A real2sim2real method for robust object grasping with neural surface reconstruction

    Luobin Wang, Runlin Guo, Quan Vuong, Yuzhe Qin, Hao Su, and Henrik Christensen. A real2sim2real method for robust object grasping with neural surface reconstruction. In2023 IEEE 19th International Conference on Automation Science and Engineering (CASE), pages 1–8. IEEE, 2023

  86. [94]

    Spar3d: Stable point-aware reconstruction of 3d objects from single images.arXiv preprint arXiv:2501.04689, 2025

    Zixuan Huang, Mark Boss, Aaryaman Vasishta, James M Rehg, and Varun Jampani. Spar3d: Stable point-aware reconstruction of 3d objects from single images.arXiv preprint arXiv:2501.04689, 2025

  87. [95]

    Pointllm: Empowering large language models to understand point clouds

    Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. InEuropean Conference on Computer Vision, pages 131–147. Springer, 2024

  88. [96]

    Point-nerf: Point-based neural radiance fields

    Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5438–5448, 2022

  89. [97]

    Pointclip: Point cloud understanding by clip

    Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8552–8562, 2022

  90. [98]

    Text2nerf: Text-driven 3d scene generation with neural radiance fields.IEEE Transactions on Visualization and Computer Graphics, 30(12):7749–7762, 2024

    Jingbo Zhang, Xiaoyu Li, Ziyu Wan, Can Wang, and Jing Liao. Text2nerf: Text-driven 3d scene generation with neural radiance fields.IEEE Transactions on Visualization and Computer Graphics, 30(12):7749–7762, 2024

  91. [99]

    Pointr: Diverse point cloud completion with geometry-aware transformers

    Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry-aware transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 12498–12507, 2021. 15

  92. [100]

    V oxel set transformer: A set-to-set approach to 3d object detection from point clouds

    Chenhang He, Ruihuang Li, Shuai Li, and Lei Zhang. V oxel set transformer: A set-to-set approach to 3d object detection from point clouds. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8417–8427, 2022

  93. [101]

    3d-vla: A 3d vision-language-action generative world model.arXiv preprint arXiv:2403.09631, 2024

    Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d vision-language-action generative world model.arXiv preprint arXiv:2403.09631, 2024

  94. [102]

    An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023

  95. [103]

    Occllama: An occupancy-language-action generative world model for autonomous driving

    Julong Wei, Shanshuai Yuan, Pengfei Li, Qingda Hu, Zhongxue Gan, and Wenchao Ding. Occllama: An occupancy-language-action generative world model for autonomous driving. arXiv preprint arXiv:2409.03272, 2024

  96. [104]

    Pointvla: Injecting the 3d world into vision-language-action models.arXiv preprint arXiv:2503.07511, 2025

    Chengmeng Li, Junjie Wen, Yan Peng, Yaxin Peng, Feifei Feng, and Yichen Zhu. Pointvla: Injecting the 3d world into vision-language-action models.arXiv preprint arXiv:2503.07511, 2025

  97. [105]

    Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics

    Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics. arXiv preprint arXiv:2411.16537, 2024

  98. [106]

    Unconditional scene graph generation

    Sarthak Garg, Helisa Dhamo, Azade Farshad, Sabrina Musatian, Nassir Navab, and Federico Tombari. Unconditional scene graph generation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 16362–16371, 2021

  99. [107]

    Incorporating structured representations into pretrained vision & language models using scene graphs.arXiv preprint arXiv:2305.06343, 2023

    Roei Herzig, Alon Mendelson, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Dar- rell, and Amir Globerson. Incorporating structured representations into pretrained vision & language models using scene graphs.arXiv preprint arXiv:2305.06343, 2023

  100. [108]

    Metricprompt: Prompting model as a relevance metric for few-shot text classification

    Hongyuan Dong, Weinan Zhang, and Wanxiang Che. Metricprompt: Prompting model as a relevance metric for few-shot text classification. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 426–436, 2023

  101. [109]

    Reasoning paths with reference objects elicit quantitative spatial reasoning in large vision-language models.arXiv preprint arXiv:2409.09788, 2024

    Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, and David Acuna. Reasoning paths with reference objects elicit quantitative spatial reasoning in large vision-language models.arXiv preprint arXiv:2409.09788, 2024

  102. [110]

    Affordancenet: An end-to-end deep learning approach for object affordance detection

    Thanh-Toan Do, Anh Nguyen, and Ian Reid. Affordancenet: An end-to-end deep learning approach for object affordance detection. In2018 IEEE international conference on robotics and automation (ICRA), pages 5882–5889. IEEE, 2018

  103. [111]

    Visual and linguistic cues to graspable objects.Experimental brain research, 229:545–559, 2013

    Andriy Myachykov, Rob Ellis, Angelo Cangelosi, and Martin H Fischer. Visual and linguistic cues to graspable objects.Experimental brain research, 229:545–559, 2013

  104. [112]

    Integrating with multimodal information for enhancing robotic grasping with vision-language models.IEEE Transactions on Automation Science and Engineering, 2025

    Zhou Zhao, Dongyuan Zheng, Yizi Chen, Jing Luo, Yanjun Wang, Panfeng Huang, and Chenguang Yang. Integrating with multimodal information for enhancing robotic grasping with vision-language models.IEEE Transactions on Automation Science and Engineering, 2025

  105. [113]

    Storm: An integrated framework for fast joint-space model-predictive control for reactive manipulation

    Mohak Bhardwaj, Balakumar Sundaralingam, Arsalan Mousavian, Nathan D Ratliff, Dieter Fox, Fabio Ramos, and Byron Boots. Storm: An integrated framework for fast joint-space model-predictive control for reactive manipulation. InConference on Robot Learning, pages 750–759. PMLR, 2022

  106. [114]

    Real-time model predictive control for industrial manipulators with singularity-tolerant hierarchical task control

    Jaemin Lee, Mingyo Seo, Andrew Bylard, Robert Sun, and Luis Sentis. Real-time model predictive control for industrial manipulators with singularity-tolerant hierarchical task control. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 12282– 12288. I...

  107. [115]

    Kinematic-model-free predictive control for robotic manipulator target reaching with obstacle avoidance.Frontiers in Robotics and AI, 9:809114, 2022

    Ahmad AlAttar, Digby Chappell, and Petar Kormushev. Kinematic-model-free predictive control for robotic manipulator target reaching with obstacle avoidance.Frontiers in Robotics and AI, 9:809114, 2022

  108. [116]

    Visual foresight: Model-based deep reinforcement learning for vision-based robotic control.arXiv preprint arXiv:1812.00568, 2018

    Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control.arXiv preprint arXiv:1812.00568, 2018. 16

  109. [117]

    Deep visual foresight for planning robot motion

    Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. InIEEE International Conference on Robotics and Automation, pages 2786–2793, 2017

  110. [118]

    Learning 3d dynamic scene represen- tations for robot manipulation.arXiv preprint arXiv:2011.01968, 2020

    Zhenjia Xu, Zhanpeng He, Jiajun Wu, and Shuran Song. Learning 3d dynamic scene represen- tations for robot manipulation.arXiv preprint arXiv:2011.01968, 2020

  111. [119]

    Object-centric forward modeling for model predictive control

    Yufei Ye, Dhiraj Gandhi, Abhinav Gupta, and Shubham Tulsiani. Object-centric forward modeling for model predictive control. InConference on Robot Learning, pages 100–109. PMLR, 2020

  112. [120]

    Learning language- conditioned robot behavior from offline data and crowd-sourced annotation

    Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation. InConference on Robot Learning, pages 1303–1315, 2022

  113. [121]

    A control-centric benchmark for video prediction

    Stephen Tian, Chelsea Finn, and Jiajun Wu. A control-centric benchmark for video prediction. InInternational Conference on Learning Representations, 2022

  114. [122]

    Sequential manipulation of deformable linear object networks with endpoint pose measurements using adaptive model predictive control

    Tyler Toner, Vahidreza Molazadeh, Miguel Saez, Dawn M Tilbury, and Kira Barton. Sequential manipulation of deformable linear object networks with endpoint pose measurements using adaptive model predictive control. In2024 IEEE International Conference on Robotics and Automation...

  115. [123]

    Obstacle avoidance shape control of deformable lin- ear objects with online parameters adaptation based on differentiable simulation.ROBOMECH Journal, 11(1):15, 2024

    Changjian Ying and Kimitoshi Yamazaki. Obstacle avoidance shape control of deformable lin- ear objects with online parameters adaptation based on differentiable simulation.ROBOMECH Journal, 11(1):15, 2024

  116. [124]

    Bimanual deformable bag manipulation using a structure-of-interest based latent dynamics model.arXiv e-prints, pages arXiv–2401, 2024

    Peng Zhou, Pai Zheng, Jiaming Qi, Chenxi Li, Chenguang Yang, David Navarro-Alarcon, and Jia Pan. Bimanual deformable bag manipulation using a structure-of-interest based latent dynamics model.arXiv e-prints, pages arXiv–2401, 2024

  117. [125]

    Semantically informed mpc for context-aware robot exploration

    Yash Goel, Narunas Vaskevicius, Luigi Palmieri, Nived Chebrolu, Kai O Arras, and Cyrill Stachniss. Semantically informed mpc for context-aware robot exploration. In2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11218–11225. IEEE, 2023

  118. [126]

    Closing the perception-action loop for semantically safe navigation in semi-static environments

    Jingxing Qian, Siqi Zhou, Nicholas Jianrui Ren, Veronica Chatrath, and Angela P Schoellig. Closing the perception-action loop for semantically safe navigation in semi-static environments. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 11641– 1164...

  119. [127]

    Automatic configuration of multi-agent model predictive controllers based on semantic graph world models

    Koen de V os, Elena Torta, Herman Bruyninckx, CA López Martínez, and MJG van de Molengraft. Automatic configuration of multi-agent model predictive controllers based on semantic graph world models. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 7...

  120. [128]

    Optimization by simulated annealing

    Scott Kirkpatrick, C Daniel Gelatt Jr, and Mario P Vecchi. Optimization by simulated annealing. science, 220(4598):671–680, 1983

  121. [129]

    Set- of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set- of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

  122. [130]

    Mixture-of-agents enhances large language model capabilities.arXiv preprint arXiv:2406.04692, 2024

    Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities.arXiv preprint arXiv:2406.04692, 2024

  123. [131]

    Seeground: See and ground for zero-shot open-vocabulary 3d visual grounding.arXiv preprint arXiv:2412.04383, 2024

    Rong Li, Shijie Li, Lingdong Kong, Xulei Yang, and Junwei Liang. Seeground: See and ground for zero-shot open-vocabulary 3d visual grounding.arXiv preprint arXiv:2412.04383, 2024

  124. [132]

    3d-llm: Injecting the 3d world into large language models.Advances in Neural Information Processing Systems, 36:20482–20494, 2023

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models.Advances in Neural Information Processing Systems, 36:20482–20494, 2023

  125. [133]

    Is a picture worth a thousand words? delving into spatial reasoning for vision language models

    Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Sharon Li, and Neel Joshi. Is a picture worth a thousand words? delving into spatial reasoning for vision language models. Advances in Neural Information Processing Systems, 37:75392–75421, 2024

  126. [134]

    On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021. 17

  127. [135]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024

  128. [136]

    Beyond semantics: Rediscovering spatial awareness in vision-language models.arXiv preprint arXiv:2503.17349, 2025

    Jianing Qi, Jiawei Liu, Hao Tang, and Zhigang Zhu. Beyond semantics: Rediscovering spatial awareness in vision-language models.arXiv preprint arXiv:2503.17349, 2025

  129. [137]

    You are a helpful UR5e robot arm t r a j e c t o r y analyzer

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. 18 A Technical Appendices and Supplementary Material A.1...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.