Pith. sign in

REVIEW 5 major objections 7 minor 79 references

T-Rex: Task-Adaptive Spatial Representation Extraction for Robotic Manipulation with Vision-Language Models

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read T-Rex claims that letting a vision-language model choose a spatial representation per task, per object, and per stage—points for simple placement, vectors for orientations, 6D poses for fine alignment—gives a manipulation system better…

desk verdict Useful system paper with a real robot gain, but the paper's headline claim—that adaptive selection, not just a bigger toolkit, drives the gain—is not actually tested. read the letter →

arxiv 2506.19498 v1 pith:XNOQTG7T submitted 2025-06-24 cs.RO cs.AI

Yiteng Chen , Wenbo Li , Shiyi Wang , Huiping Zhuang , Qingyao Wu This is my paper
classification cs.ROcs.AI
keywords task-adaptivespatialrepresentationextractionvision-languagemodelsforroboticsroboticmanipulationChainofGrounding6Dposeestimationkeypointzero-shotrobotcontrolconstraintgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a robot manipulation system guided by a vision-language model should not commit to one spatial representation—points, vectors, or 6D poses—for every task. Instead, it should pick the type and granularity of representation per object and per task stage, because simple tasks only need a center point while orientation-sensitive tasks need a full 6D pose, and richer representations cost more time. The paper's system, T-Rex, operationalizes this by letting the VLM choose tools from an extensible registry under a step-by-step Chain of Grounding prompt, then generating Python constraint functions and a low-level action sequence. In 150 real-world trials across 15 everyday tasks, it reports 60.7% success versus 36.4% for a fixed keypoint-constraint baseline and 30% for a value-map baseline, with a lower average completion time. The central claim is that the adaptive selection mechanism itself, not any single extractor, delivers the gain, and all of this happens without additional training.

What carries the argument

The load-bearing mechanism is the pairing of two components: an extensible Spatial Representation Extraction Toolkit—a registry of extractors (center points, semantic keypoints, uniform scene keypoints, 6D pose, local subimage crops, and VLM-generated point, vector, state-machine, and topology tools), each annotated with inputs, output format, and historical average execution time—and the Chain of Grounding (CoG) prompt structure that takes a natural-language instruction and a scene image and forces the VLM through four phases: operational hints, natural-language constraints, per-object tool selection, and Python constraint-code generation. The formal selection criterion is $t_{s,o}^{*}=\arg\max_{t\in\mathcal{R}}\left[P_{\mathrm{succ}}(t\mid I,X,s,o)-\lambda h_{t}\right]$, balancing estimated extraction success against average running time; the paper notes that in practice the VLM is prompted to choose the simplest tool whose extracted representation satisfies the task requirements. This mechanism produces heterogeneous, multi-granular spatial representations that are then consumed by a solver or by generated policy code to yield end-effector trajectories.

What would settle it

Run the same 15 tasks on the same robot with the same toolkit and the same Chain of Grounding prompts, but replace the adaptive tool selection with each single fixed extractor (points only, vectors only, 6D poses only) and check whether adaptive selection still beats the best fixed configuration. Since the paper's own ablation shows a fixed simplest point extractor drops success to 30.7 percent and a fixed point-and-vector extractor to 55.0 percent, a narrower observed gap would indicate the advantage comes from the toolkit's stronger extractors rather than from adaptation.

Watch

Extended reading notes

Core claim

T-Rex's central discovery is that task complexity determines the type and granularity of spatial representation needed, and that stronger representational power carries higher extraction cost; therefore choosing the simplest sufficient extractor for each entity at each stage yields both better spatial understanding and better efficiency. Concretely, the framework maintains an extensible toolkit registry of extractors, and the Chain of Grounding prompt makes the VLM decompose the instruction into stages, infer operational hints, select a tool for every task-relevant object (preferring the simplest that satisfies the requirement), and emit per-stage Python constraint functions. The resulting heterogeneous, multi-granular representations feed a solver or a generated policy script to produce end-effector trajectories. The paper reports 60.7% average success over 150 trials (15 tasks times 10 trials) compared with 36.4% for the automatic keypoint-constraint baseline and 30% for the value-map baseline, while taking 45.5 seconds on average versus 53.3 and 50.8 seconds. It concludes that the adaptive selection mechanism—supported by Chain of Grounding—is the source of the improvement, not any individual extractor.

Load-bearing premise

The whole system succeeds only if a proprietary vision-language model, prompted by the Chain of Grounding routine, reliably decomposes instructions, chooses the right extractor from the registry, and writes correct Python constraint functions with no training—if that model or prompt fails, the claimed advantage collapses.

Editorial extensions

If this is right

  • New extractors, including tactile or future pose-estimation modules, can be added to the registry and become available to the planner without retraining the system.
  • Tasks that mix coarse positioning with fine orientation requirements, such as placing plush toys upright and aligned, can use cheap point representations for positioning and 6D poses only where orientation matters, avoiding the time cost of full-pose extraction on every object.
  • Because the toolkit chooses the simplest sufficient tool, the paper's average completion time over the 15 tasks falls to 45.5 seconds, about 8 seconds faster than the keypoint-constraint baseline and 5 seconds faster than the value-map baseline.
  • Removing Chain of Grounding drops success to 52.1%, so the explicit step-by-step grounding of instructions into hints, constraints, tool choices, and code is load-bearing for the method's stability.
  • The system claims plug-and-play deployment with no additional training, meaning its performance ceiling is set by the reasoning quality of the vision-language model and the quality of registered extractors, not by robot-specific data collection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same selection logic could be ported to other VLM-driven constraint-manipulation pipelines, which suggests the transferable contribution is the selection policy itself, not the particular extractors in the toolkit.
  • Beyond the paper: because the formal argmax criterion is approximated by a 'choose the simplest tool' prompt, a learned success-probability estimator trained on logged tool executions could make the method dependable with weaker or cheaper VLMs.
  • Beyond the paper: the ablation spread—30.7 percent with a fixed point extractor, 55.0 percent with a fixed point-and-vector extractor, 60.7 percent with adaptation—predicts that adaptation matters most on tasks mixing coarse positioning with fine orientation requirements, and least on purely positional tasks.
  • Beyond the paper: because the paper reports success varying from 55.0 to 61.4 percent across different vision-language models, it implies that the current ceiling is set by model inference quality; stronger spatial reasoning in future VLMs should raise T-Rex's ceiling without changing its architecture.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes T-Rex, a framework for real-world robotic manipulation in which a Vision-Language Model (GPT-4.1) grounds a natural-language instruction through a four-stage Chain of Grounding (CoG), selects per-object, per-stage spatial representation extractors from an extensible toolkit (keypoints, vectors, 6D pose, local sub-image, state machines, topological orderings), and generates Python constraint functions that a low-level solver converts into end-effector trajectories. The main empirical claim is that task-adaptive selection of representation types and granularities yields better spatial understanding, efficiency, and stability than fixed-representation baselines VoxPoser and ReKep, with 60.7% success over 150 real-world trials in 15 tasks versus 36.4% and 30%, and lower average completion time (45.5 s versus 53.3 s and 50.8 s), all without additional training.

Significance. If the central claim is established, T-Rex would be a useful contribution: it is training-free, modular, and the reported aggregate gains over two fixed-representation baselines are large and consistent across several task categories. The paper also provides a useful decomposition of error sources (Fig. 6), a VLM-choice study (Table 3), and an honest discussion of limitations and dependencies on proprietary models. However, the experiments as reported do not yet isolate the adaptive selection mechanism from the contents of the toolkit, the formal selection criterion in Sec. 3.2.1 is not operationalized, the efficiency numbers in Table 1 are internally inconsistent, and the claimed multi-granularity ablation is missing. These issues leave the paper's strongest claim underdetermined rather than refuted.

major comments (5)
  1. [Sec. 4.2, Table 2] The central claim that task-adaptive selection drives the improvement is not isolated by the ablations. The 'w/o Toolkit (SP)' and 'w/o Toolkit (VPV)' conditions remove the entire toolkit and replace it with one fixed extractor, so they change both the selection policy and the set of available tools; the 5.7-point gap between Ours and w/o Toolkit (VPV) could be due to the added SE3 pose and multi-granularity extractors rather than to selecting among them per task. Please add a control that holds the toolkit fixed and varies only the selection policy (e.g., always use the most capable tool, or a random or oracle selector), or soften the causal claim accordingly.
  2. [Sec. 3.2.1, Eq. (1); Appendix A.4] The formal tool-selection criterion t* = argmax [P_succ - lambda*h_t] is not operationalized anywhere. P_succ is never estimated, lambda is never set, and Appendix A.4 instead states that the VLM is prompted to choose 'the simplest tool' whose representation satisfies the task. This is not merely a presentation gap: the paper's efficiency-accuracy trade-off claim depends on that criterion. Please either specify how P_succ and lambda are computed or calibrated, or revise the formalization to describe the heuristic actually used.
  3. [Sec. 3.2.2; Sec. 4.2] The text in Sec. 3.2.2 says that fine-grained extraction from the raw view 'frequently fail[s]... as evidenced by our ablation studies in Sec. 4.2,' but Table 2 contains no row that toggles the multi-granularity (local sub-image) mechanism while keeping the rest of the pipeline fixed. The promised ablation is absent, so the necessity of multi-granularity extraction is not supported by the reported data. Please add the ablation or remove the reference.
  4. [Table 1] The reported total average completion times do not match the arithmetic means of the per-task times. For T-Rex the per-task times sum to 637.4 s across 15 tasks, giving a mean of 42.5 s, not 45.5 s; analogous discrepancies appear for VoxPoser (47.4 vs 50.8) and ReKep (49.7 vs 53.3). Since efficiency is one of the three headline claims, please clarify how the total times are computed (e.g., including setup time, conditioning on success, or using different trial sets) and correct any inconsistency.
  5. [Sec. 4.1, Table 1] All per-task results are based on 10 trials, and no significance testing or confidence intervals are reported for either success rates or completion times. With n = 10, several per-task comparisons are within the range of sampling noise (e.g., Stack Block 6/10 vs 6/10; Sweep Trash 9/10 vs 8/10), and the claim of 'significant advantages' needs at least exact binomial or bootstrap intervals. Adding this analysis does not require new experiments, but the statistical support should match the strength of the wording.
minor comments (7)
  1. [Sec. 5 / Conclusion] The conclusion says 'Extensive experiments in both simulated and real-world robotic environments,' but the paper reports only real-world experiments. Please correct the statement or add simulation results.
  2. [Sec. 3.2 heading; throughout] There are typographical errors such as 'Rpresentation' in the Sec. 3.2 heading and 'V oxPoser' with stray spaces in several places. Please proofread.
  3. [Appendix A.5.2] The method SPAR3D is written 'SRAR3D' in one sentence. Please unify the spelling.
  4. [Appendix A.5.1, Listing 3] The registry entries are not valid JSON: several entries are missing commas or closing braces (e.g., the Local SubImage Extractor and VLM Task Vector Extractor entries). If the registry is meant to be machine-readable, provide a validated example.
  5. [Fig. 6 and main text] Stray annotations such as 'zi too small' and 'setup 1 duan' appear in or near Figure 6 and the experimental text. These should be removed or fixed before submission.
  6. [Table 3] The 'Inference Validity Rate' is human-evaluated, but the evaluation protocol, the number of annotators, and the exact rubric are not described. Please add details.
  7. [Appendix A.9] There is a typo: 'seletions' should be 'selections'. Also, the notation for the CoG mapping is introduced twice (Sec. 3.3 and A.9) with slightly different formatting; please standardize.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: T-Rex's success is measured by external physical task completion, and no fitted parameter is relabeled as a prediction.

full rationale

T-Rex is an engineered system rather than a derived result, and its evaluation is anchored to an external criterion. The reported success rates are physical task completions with hand-defined criteria (Appendix A.1.2), not judgments produced by the system's own VLM constraints, so the evaluation does not reduce to the model's self-assessment. The formal selection rule in Sec. 3.2.1 (argmax over P_succ - lambda * h_t) is not operationalized, and Appendix A.4 replaces it with a prompt asking the VLM to choose 'the simplest tool' whose representation satisfies the task; this is an implementation gap, not a circular reduction, because the paper never claims P_succ was estimated or that Eq. 1 was used to produce the headline numbers. The ablation 'w/o Toolkit' removes the entire toolkit rather than only the adaptive selection policy, confounding adaptivity with tool availability, but confounding and underdetermination are not equivalence-by-construction; no equation in the paper makes the reported advantage equal to its inputs. There are no load-bearing self-citations: the only substantial external component (solver-based action generation) is explicitly credited to ReKep and disclaimed as a contribution. No fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work. The central claim is therefore not circular; it is under-supported by the ablations, which is a correctness and experimental-design concern rather than a circularity finding.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical or ontological entities are postulated. Chain of Grounding, the Toolkit registry, and the state-machine/topology representations are software or conceptual constructs built from existing models. The main unstated cost is the reliability of proprietary VLM and vision-tool components, captured in the axioms.

free parameters (2)
  • lambda (cost-efficiency trade-off in tool selection) = not specified
    Appears in the formal criterion t* = argmax [P_succ - lambda*h_t] in Sec. 3.2.1, but no value is given and the criterion is not actually used; tool selection is performed by VLM prompting.
  • Local crop padding scale = not specified
    Sec. 3.2.2 invokes a fixed or adaptive padding to expand the SAM mask before cropping; the choice affects fine-grained extraction quality and is not quantified.
assumptions (4)
  • domain assumption External vision tools (Grounding DINO, SAM, DINOv2, SAM6D, SPAR3D) return sufficiently accurate detections, keypoints, and 6D poses when invoked by the toolkit.
    The whole system is a pipeline of off-the-shelf models; no calibration or error bounds are reported. The paper itself notes SPAR3D meshes can have incomplete backside reconstruction (A.5.2).
  • domain assumption GPT-4.1 with CoG prompts provides reliable zero-shot constraint generation and tool selection.
    All task success flows through VLM outputs. Table 3 shows model choice shifts success by several points, and Sec. 3.3 states that without CoG the VLM frequently produces incomplete or incorrect constraints or chooses clearly unsuitable extraction tools.
  • domain assumption Stronger representational capability implies higher extraction cost.
    Stated as the key insight in the abstract and Sec. 3.2; it motivates the trade-off but is not independently measured or derived.
  • domain assumption The baselines (VoxPoser, ReKep Auto) were implemented and configured fairly.
    Only a brief consistency statement in A.1.3 is provided; no code or detailed hyperparameters for the baselines are given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of T-Rex: Task-Adaptive Spatial Representation Extraction for Robotic Manipulation with Vision-Language Models." pith.science (2026). https://pith.science/paper/XNOQTG7T

@misc{pith2026250619498,
  author       = {Pith},
  title        = {Pith review of: T-Rex: Task-Adaptive Spatial Representation Extraction for Robotic Manipulation with Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XNOQTG7T}},
  note         = {Machine review of arXiv:2506.19498}
}
read the original abstract

Building a general robotic manipulation system capable of performing a wide variety of tasks in real-world settings is a challenging task. Vision-Language Models (VLMs) have demonstrated remarkable potential in robotic manipulation tasks, primarily due to the extensive world knowledge they gain from large-scale datasets. In this process, Spatial Representations (such as points representing object positions or vectors representing object orientations) act as a bridge between VLMs and real-world scene, effectively grounding the reasoning abilities of VLMs and applying them to specific task scenarios. However, existing VLM-based robotic approaches often adopt a fixed spatial representation extraction scheme for various tasks, resulting in insufficient representational capability or excessive extraction time. In this work, we introduce T-Rex, a Task-Adaptive Framework for Spatial Representation Extraction, which dynamically selects the most appropriate spatial representation extraction scheme for each entity based on specific task requirements. Our key insight is that task complexity determines the types and granularity of spatial representations, and Stronger representational capabilities are typically associated with Higher overall system operation costs. Through comprehensive experiments in real-world robotic environments, we show that our approach delivers significant advantages in spatial understanding, efficiency, and stability without additional training.

Figures

Figures reproduced from arXiv: 2506.19498 by the authors.

Figure 1
Figure 1. T-Rex adaptively invokes the optimal extractor from the Spatial Representation Extraction [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview framework. Given natural language instruction and scene observation, VLM [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Spatial Representation Extraction Toolkit. System adaptively invokes the optimal extractor [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Multi-Granularity spatial representation extraction. First crop a local subgraph, then extract [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Because T-Rex Toolkit includes 6D pose estimation module, it outperforms the keypoint [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: System error breakdown.zi too small The ablation results are summarized in Tab. 2. We first observe that integrating CoG yields a clear improvement in success rate with insignificant additional latency. Although the improvement is modest, CoG consistently delivers a st…
Figure 7
Figure 7. Figure 7: Single-Arm Platform. As shown in [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Execution Snapshots for Sample Tasks 18 [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: An example of a State Machine. Modeling the drawer’s states to support the generation of [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: Example experimental props and the 3D meshes we reconstructed [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

79 extracted references · 22 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation

    Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652, 2024

    arXiv 2024

  3. [3]

    Copa: General robotic manipulation through spatial constraints of parts with foundation models

    Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9488–9495. IEEE, 2024

    work page 2024

  4. [4]

    Omni- manip: Towards general robotic manipulation via object-centric interaction primitives as spatial constraints

    Mingjie Pan, Jiyao Zhang, Tianshu Wu, Yinghao Zhao, Wenlong Gao, and Hao Dong. Omni- manip: Towards general robotic manipulation via object-centric interaction primitives as spatial constraints. arXiv preprint arXiv:2501.03841, 2025

    arXiv 2025

  5. [5]

    Geomanip: Geometric constraints as general interfaces for robot manipulation

    Weiliang Tang, Jia-Hui Pan, Yun-Hui Liu, Masayoshi Tomizuka, Li Erran Li, Chi-Wing Fu, and Mingyu Ding. Geomanip: Geometric constraints as general interfaces for robot manipulation. arXiv preprint arXiv:2501.09783, 2025

    arXiv 2025

  6. [6]

    Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning

    Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao. Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning. arXiv preprint arXiv:2311.17842, 2023

    arXiv 2023

  7. [7]

    Guiding long-horizon task and motion planning with vision language models

    Zhutian Yang, Caelan Garrett, Dieter Fox, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. Guiding long-horizon task and motion planning with vision language models. arXiv preprint arXiv:2410.02193, 2024

    arXiv 2024

  8. [8]

    Open-world task and mo- tion planning via vision-language model inferred constraints

    Nishanth Kumar, Fabio Ramos, Dieter Fox, and Caelan Reed Garrett. Open-world task and mo- tion planning via vision-language model inferred constraints. arXiv preprint arXiv:2411.08253, 2024

    arXiv 2024

Show all 79 references
  1. [9]

    Physically grounded vision-language models for robotic manipulation

    Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. Physically grounded vision-language models for robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 12462–12469. IEEE, 2024

    2024

  2. [10]

    Vlm see, robot do: Human demo video to robot action plan via vision language model

    Beichen Wang, Juexiao Zhang, Shuwen Dong, Irving Fang, and Chen Feng. Vlm see, robot do: Human demo video to robot action plan via vision language model. arXiv preprint arXiv:2410.08792, 2024

    2024

  3. [11]

    Sofar: Language-grounded orientation bridges spatial reasoning and object manipulation

    Zekun Qi, Wenyao Zhang, Yufei Ding, Runpei Dong, Xinqiang Yu, Jingwen Li, Lingyun Xu, Baoyu Li, Xialin He, Guofan Fan, et al. Sofar: Language-grounded orientation bridges spatial reasoning and object manipulation. arXiv preprint arXiv:2502.13143, 2025

    2025

  4. [12]

    Vlmpc: Vision-language model predictive control for robotic manipulation

    Wentao Zhao, Jiaming Chen, Ziyu Meng, Donghui Mao, Ran Song, and Wei Zhang. Vlmpc: Vision-language model predictive control for robotic manipulation. arXiv preprint arXiv:2407.09829, 2024

    2024 arXiv

  5. [13]

    Kuda: Keypoints to unify dynamics learning and visual prompting for open-vocabulary robotic manipulation

    Zixian Liu, Mingtong Zhang, and Yunzhu Li. Kuda: Keypoints to unify dynamics learning and visual prompting for open-vocabulary robotic manipulation. arXiv preprint arXiv:2503.10546, 2025

    2025 arXiv

  6. [14]

    V oxposer: Composable 3d value maps for robotic manipulation with language models

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023

    2023 arXiv

  7. [15]

    Moka: Open-world robotic manipulation through mark-based visual prompting

    Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-world robotic manipulation through mark-based visual prompting. arXiv preprint arXiv:2403.03174, 2024. 10

    2024 arXiv

  8. [16]

    A real-to-sim-to-real approach to robotic manipulation with vlm-generated iterative keypoint rewards

    Shivansh Patel, Xinchen Yin, Wenlong Huang, Shubham Garg, Hooshang Nayyeri, Li Fei-Fei, Svetlana Lazebnik, and Yunzhu Li. A real-to-sim-to-real approach to robotic manipulation with vlm-generated iterative keypoint rewards. arXiv preprint arXiv:2502.08643, 2025

    2025 arXiv

  9. [17]

    Learning to interpret natural language commands through human-robot dialog

    Jesse Thomason, Shiqi Zhang, Raymond J Mooney, and Peter Stone. Learning to interpret natural language commands through human-robot dialog. In IJCAI, volume 15, pages 1923– 1929, 2015

    1923

  10. [18]

    Grounding verbs of motion in natural language commands to robots

    Thomas Kollar, Stefanie Tellex, Deb Roy, and Nicholas Roy. Grounding verbs of motion in natural language commands to robots. In Experimental robotics: The 12th international symposium on experimental robotics, pages 31–47. Springer, 2014

    2014

  11. [19]

    Toward understanding natural language directions

    Thomas Kollar, Stefanie Tellex, Deb Roy, and Nicholas Roy. Toward understanding natural language directions. In 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 259–266. IEEE, 2010

    2010

  12. [20]

    Understanding natural language commands for robotic navigation and mobile manipulation

    Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI conference on artificial intelligence , volume 25...

    2011

  13. [21]

    Tinyvla: Towards fast, data-efficient vision-language-action models for robotic manipulation

    Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, Yaxin Peng, et al. Tinyvla: Towards fast, data-efficient vision-language-action models for robotic manipulation. arXiv preprint arXiv:2409.12514, 2024

    2024 arXiv

  14. [22]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024

    2024 arXiv

  15. [23]

    Rdt-1b: a diffusion foundation model for bimanual manipulation

    Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864, 2024

    2024 arXiv

  16. [24]

    Rt-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022

    2022 arXiv

  17. [25]

    Rt-2: Vision-language- action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language- action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023

    2023 arXiv

  18. [26]

    Rt-trajectory: Robotic task generalization via hindsight trajectory sketches

    Jiayuan Gu, Sean Kirmani, Paul Wohlhart, Yao Lu, Montserrat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, et al. Rt-trajectory: Robotic task generalization via hindsight trajectory sketches. arXiv preprint arXiv:2311.01977, 2023

    2023 arXiv

  19. [27]

    Aloha 2: An enhanced low-cost hardware for bimanual teleoperation

    Jorge Aldaco, Travis Armstrong, Robert Baruch, Jeff Bingham, Sanky Chan, Kenneth Draper, Debidatta Dwibedi, Chelsea Finn, Pete Florence, Spencer Goodrich, et al. Aloha 2: An enhanced low-cost hardware for bimanual teleoperation. arXiv preprint arXiv:2405.02292, 2024

    2024 arXiv

  20. [28]

    Mobile aloha: Learning bimanual mobile manipu- lation with low-cost whole-body teleoperation

    Zipeng Fu, Tony Z Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipu- lation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02117, 2024

    2024 arXiv

  21. [29]

    Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation

    Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al. Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation. arXiv preprint arXiv:2411.19650, 2024

    2024 arXiv

  22. [30]

    3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations

    Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. arXiv preprint arXiv:2403.03954, 2024. 11

    2024 arXiv

  23. [31]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213, 2024

    2024 arXiv

  24. [32]

    Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation

    Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, et al. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158, 2024

    arXiv

  25. [33]

    Roboflamingo-plus: Fusion of depth and rgb perception with vision-language models for enhanced robotic manipulation

    Sheng Wang. Roboflamingo-plus: Fusion of depth and rgb perception with vision-language models for enhanced robotic manipulation. arXiv preprint arXiv:2503.19510, 2025

    2025 arXiv

  26. [34]

    Vision-language foundation models as effective robot imitators

    Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators. arXiv preprint arXiv:2311.01378, 2023

    2023 arXiv

  27. [35]

    Hybridvla: Collaborative diffusion and autoregression in a unified vision-language-action model

    Jiaming Liu, Hao Chen, Pengju An, Zhuoyang Liu, Renrui Zhang, Chenyang Gu, Xiaoqi Li, Ziyu Guo, Sixiang Chen, Mengzhen Liu, et al. Hybridvla: Collaborative diffusion and autoregression in a unified vision-language-action model. arXiv preprint arXiv:2503.10631, 2025

    2025 arXiv

  28. [36]

    Cot-vla: Visual chain-of-thought reasoning for vision-language-action models

    Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, et al. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. arXiv preprint arXiv:2503.22020, 2025

    2025 arXiv

  29. [37]

    Pointvla: Injecting the 3d world into vision-language-action models

    Chengmeng Li, Junjie Wen, Yan Peng, Yaxin Peng, Feifei Feng, and Yichen Zhu. Pointvla: Injecting the 3d world into vision-language-action models. arXiv preprint arXiv:2503.07511, 2025

    2025 arXiv

  30. [38]

    Spatialvla: Exploring spatial representations for visual-language-action model

    Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, JiaYuan Gu, Bin Zhao, Dong Wang, et al. Spatialvla: Exploring spatial representations for visual-language-action model. arXiv preprint arXiv:2501.15830, 2025

    2025 arXiv

  31. [39]

    Hi robot: Open-ended instruction following with hierarchical vision-language-action models

    Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, et al. Hi robot: Open-ended instruction following with hierarchical vision-language-action models. arXiv preprint arXiv:2502.19417, 2025

    2025 arXiv

  32. [40]

    Gr00t n1: An open foundation model for generalist humanoid robots

    Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025

    2025 arXiv

  33. [41]

    Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents

    Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, et al. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. arXiv preprint arX...

    2025 arXiv

  34. [42]

    Llm+ p: Empowering large language models with optimal planning proficiency

    Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477, 2023

    2023 arXiv

  35. [43]

    Llm-planner: Few-shot grounded planning for embodied agents with large language models

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2998–3009, 2023

    2023

  36. [44]

    Code as policies: Language model programs for embodied control

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493–9500. IEEE, 2023

    2023

  37. [45]

    Progprompt: Generating situated robot task plans using large language models

    Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automatio...

    2023

  38. [46]

    Chatgpt for robotics: Design principles and model abilities

    Sai H Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. Ieee Access, 2024

    2024

  39. [47]

    Toward general-purpose robots via foundation models: A survey and meta-analysis

    Yafei Hu, Quanting Xie, Vidhi Jain, Jonathan Francis, Jay Patrikar, Nikhil Keetha, Seungchan Kim, Yaqi Xie, Tianyi Zhang, Hao-Shu Fang, et al. Toward general-purpose robots via foundation models: A survey and meta-analysis. arXiv preprint arXiv:2312.08782, 2023

    2023 arXiv

  40. [48]

    Foundation models defining a new era in vision: a survey and outlook

    Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Foundation models defining a new era in vision: a survey and outlook. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  41. [49]

    Yolov10: Real-time end-to-end object detection

    Ao Wang, Hui Chen, Lihao Liu, Kai Chen, Zijia Lin, Jungong Han, et al. Yolov10: Real-time end-to-end object detection. Advances in Neural Information Processing Systems, 37:107984– 108011, 2024

    2024

  42. [50]

    Yolov12: Attention-centric real-time object detectors

    Yunjie Tian, Qixiang Ye, and David Doermann. Yolov12: Attention-centric real-time object detectors. arXiv preprint arXiv:2502.12524, 2025

    2025 arXiv

  43. [51]

    Yoloe: Real-time seeing anything

    Ao Wang, Lihao Liu, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. Yoloe: Real-time seeing anything. arXiv preprint arXiv:2503.07465, 2025

    2025

  44. [52]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pages 38–55. Springer, 2024

    2024

  45. [53]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024

    2024 arXiv

  46. [54]

    Fast segment anything

    Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment anything. arXiv preprint arXiv:2306.12156, 2023

    2023 arXiv

  47. [55]

    Segment everything everywhere all at once

    Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in neural information processing systems, 36:19769–19782, 2023

    2023

  48. [56]

    kpam: Keypoint affordances for category-level robotic manipulation

    Lucas Manuelli, Wei Gao, Peter Florence, and Russ Tedrake. kpam: Keypoint affordances for category-level robotic manipulation. In The International Symposium of Robotics Research, pages 132–157. Springer, 2019

    2019

  49. [57]

    Any-point trajectory modeling for policy learning

    Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025, 2023

    2023 arXiv

  50. [58]

    Pivot: Iterative visual prompting elicits actionable knowledge for vlms

    Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07872, 2024

    2024 arXiv

  51. [59]

    Foundationpose: Unified 6d pose estimation and tracking of novel objects

    Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17868–17879, 2024

    2024

  52. [60]

    Sam-6d: Segment anything model meets zero-shot 6d object pose estimation

    Jiehong Lin, Lihua Liu, Dekun Lu, and Kui Jia. Sam-6d: Segment anything model meets zero-shot 6d object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27906–27916, 2024

    2024

  53. [61]

    Omni6dpose: A benchmark and model for universal 6d object pose estimation and tracking

    Jiyao Zhang, Weiyao Huang, Bo Peng, Mingdong Wu, Fei Hu, Zijian Chen, Bo Zhao, and Hao Dong. Omni6dpose: A benchmark and model for universal 6d object pose estimation and tracking. In European Conference on Computer Vision, pages 199–216. Springer, 2024

    2024

  54. [62]

    Gen6d: Generalizable model-free 6-dof object pose estimation from rgb images

    Yuan Liu, Yilin Wen, Sida Peng, Cheng Lin, Xiaoxiao Long, Taku Komura, and Wenping Wang. Gen6d: Generalizable model-free 6-dof object pose estimation from rgb images. In European Conference on Computer Vision, pages 298–315. Springer, 2022. 13

    2022

  55. [63]

    Onepose: One-shot object pose estimation without cad models

    Jiaming Sun, Zihao Wang, Siyu Zhang, Xingyi He, Hongcheng Zhao, Guofeng Zhang, and Xiaowei Zhou. Onepose: One-shot object pose estimation without cad models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6825–6834, 2022

    2022

  56. [64]

    Onepose++: Keypoint-free one-shot object pose estimation without cad models

    Xingyi He, Jiaming Sun, Yuang Wang, Di Huang, Hujun Bao, and Xiaowei Zhou. Onepose++: Keypoint-free one-shot object pose estimation without cad models. Advances in Neural Infor- mation Processing Systems, 35:35103–35115, 2022

    2022

  57. [65]

    Gs-pose: Generalizable segmentation-based 6d object pose estimation with 3d gaussian splatting

    Dingding Cai, Janne Heikkilä, and Esa Rahtu. Gs-pose: Generalizable segmentation-based 6d object pose estimation with 3d gaussian splatting. arXiv preprint arXiv:2403.10683, 2024

    2024 arXiv

  58. [66]

    You only demonstrate once: Category-level manipulation from single visual demonstration.arXiv preprint arXiv:2201.12716, 2022

    Bowen Wen, Wenzhao Lian, Kostas Bekris, and Stefan Schaal. You only demonstrate once: Category-level manipulation from single visual demonstration.arXiv preprint arXiv:2201.12716, 2022

    2022 arXiv

  59. [67]

    Seeground: See and ground for zero-shot open-vocabulary 3d visual grounding

    Rong Li, Shijie Li, Lingdong Kong, Xulei Yang, and Junwei Liang. Seeground: See and ground for zero-shot open-vocabulary 3d visual grounding. arXiv preprint arXiv:2412.04383, 2024

    2024 arXiv

  60. [68]

    Vlm- grounder: A vlm agent for zero-shot 3d visual grounding

    Runsen Xu, Zhiwei Huang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Vlm- grounder: A vlm agent for zero-shot 3d visual grounding. arXiv preprint arXiv:2410.13860, 2024

    2024 arXiv

  61. [69]

    Tsp3d: Text- guided sparse voxel pruning for efficient 3d visual grounding

    Wenxuan Guo, Xiuwei Xu, Ziwei Wang, Jianjiang Feng, Jie Zhou, and Jiwen Lu. Tsp3d: Text- guided sparse voxel pruning for efficient 3d visual grounding. arXiv preprint arXiv:2502.10392, 2025

    2025 arXiv

  62. [70]

    Scanreason: Empowering 3d visual grounding with reasoning capabilities

    Chenming Zhu, Tai Wang, Wenwei Zhang, Kai Chen, and Xihui Liu. Scanreason: Empowering 3d visual grounding with reasoning capabilities. In European Conference on Computer Vision, pages 151–168. Springer, 2024

    2024

  63. [71]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  64. [72]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    2023 arXiv

  65. [73]

    Spar3d: Stable point-aware reconstruction of 3d objects from single images

    Zixuan Huang, Mark Boss, Aaryaman Vasishta, James M Rehg, and Varun Jampani. Spar3d: Stable point-aware reconstruction of 3d objects from single images. arXiv preprint arXiv:2501.04689, 2025

    2025 arXiv

  66. [74]

    9dtact: A compact vision-based tactile sensor for accurate 3d shape reconstruction and generalizable 6d force estimation

    Changyi Lin, Han Zhang, Jikai Xu, Lei Wu, and Huazhe Xu. 9dtact: A compact vision-based tactile sensor for accurate 3d shape reconstruction and generalizable 6d force estimation. IEEE Robotics and Automation Letters, 9(2):923–930, 2023

    2023

  67. [75]

    Tac3d: A novel vision-based tactile sensor for measuring forces distribution and estimating friction coefficient distribution

    Lunwei Zhang, Yue Wang, and Yao Jiang. Tac3d: A novel vision-based tactile sensor for measuring forces distribution and estimating friction coefficient distribution. arXiv preprint arXiv:2202.06211, 2022

    2022 arXiv

  68. [76]

    Pointodyssey: A large-scale synthetic dataset for long-term point tracking

    Yang Zheng, Adam W Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19855–19865, 2023

    2023

  69. [77]

    Cotracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. In European Conference on Computer Vision, pages 18–35. Springer, 2024

    2024

  70. [78]

    Robotap: Tracking arbitrary points for few-shot visual imitation

    Mel Vecerik, Carl Doersch, Yi Yang, Todor Davchev, Yusuf Aytar, Guangyao Zhou, Raia Hadsell, Lourdes Agapito, and Jon Scholz. Robotap: Tracking arbitrary points for few-shot visual imitation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5397–5...

    2024

  71. [79]

    Recyclable

    Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20406–20417, 2024. 15 A Technical Append...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.