Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Language-Conditioned Open-Vocabulary Mobile Manipulation with Pretrained Models

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Zero-shot mobile manipulation hits 10.2% on unseen tasks

desk verdict A useful modular integration of GPT-4 parsing, VLMaps navigation, and a 6-DoF CLIPort extension, plus a new benchmark; the system beats its direct baselines, but the headline zero-shot claim outruns the evidence because navigation is never measured separately. read the letter →

arxiv 2507.17379 v1 pith:SXMGO4WU submitted 2025-07-23 cs.RO

Shen Tan , Dong Zhou , Xiangyu Shao , Junqiao Wang , Guanghui Sun This is my paper
classification cs.RO
keywords open-vocabularymobilemanipulationlanguage-conditionedzero-shotgeneralization6-DoFvision-languagemodelslargelanguageimitationlearningmulti-task
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a household mobile robot can take a free-form sentence like "toss the food boxes on the office room desk to the grey trash bin in the corner" and carry it out without having seen that exact task, object, or room during training. To do this it assembles a pipeline: a large language model splits the instruction into a destination workspace and a manipulation description, a vision-language 3D map localizes the workspace so the robot can navigate there, and a two-stream imitation network trained on pick-and-place demonstrations predicts 6-DoF manipulation poses from RGB-D images plus the language description. The central quantitative claim is a 10.2% average task success rate across eight unseen OVMM tasks, and a 12.9% average on unseen tabletop manipulation tasks with 100 demonstrations, against 3.6% for CLIPort and 1.1% for the 6-DoF Transporter baseline. If the claim holds, it supports a recipe: rather than training one monolithic model for whole mobile-manipulation tasks, compose pretrained language, mapping, and manipulation components and train only the manipulation policy.

What carries the argument

The load-bearing mechanism is the two-stream end-to-end manipulation network. One stream is a pretrained CLIP ResNet-50 image encoder with skip-connected upsampling, whose feature maps are multiplied elementwise by a tiled CLIP text embedding of the manipulation description; the other stream is a Transporter-style hourglass ResNet that encodes RGB-D spatial structure. Lateral connections fuse the semantic and spatial tensors, an FCN outputs a pick affordance, and a cropped pick feature is cross-correlated with a place feature map to score SE(2) place candidates; three MLP heads then predict roll, pitch, and height from the fused features, turning 2D template matching into 6-DoF prediction. Around this sit the two pretrained parsers: GPT-4 splits the instruction, and a VLMaps-style map built from LSeg pixel embeddings is used for open-vocabulary localization and navigation. The claim depends on each stage; the trainable part is the manipulation policy, trained by imitation with cross-entropy on pick/place maps and a Huber loss on the remaining degrees of freedom.

What would settle it

Run the same eight unseen OVMM tasks with the robot's target workspace coordinates supplied directly, bypassing map-based localization while keeping language parsing and manipulation; if the per-task success rates do not rise well above the reported values, the navigation stage is not the bottleneck, whereas a large rise would show that the end-to-end manipulation model is not the sole source of the reported 10.2% success rate.

Watch

Extended reading notes

Core claim

The paper's claim is that open-vocabulary mobile manipulation can be decomposed into navigation and tabletop manipulation subtasks, solved by a single language-conditioned manipulation model in combination with pretrained parsing and mapping components. The framework, LOVMM, is presented as the first system that, given only free-form natural language, zero-shot completes diverse OVMM tasks across separate workspaces in simulated household scenes while also beating recent vision-based tabletop manipulation models. The evidence is the average 10.2% task success rate on eight unseen tasks (individual tasks range from 3.2% to 21.2%) and the 12.9% versus 3.6% versus 1.1% average success at 100 demonstrations on tabletop tasks. The paper also claims efficiency: one demonstration per seen task yields 53.4% average success on seen OVMM tasks, and 100 demonstrations yield 60.7%.

Load-bearing premise

The reported end-to-end success rates assume that the language parser and the vision-language navigation map work reliably in the unseen scenes; if either fails at a nontrivial rate, part of the 10.2% average is a navigation or parsing failure, not a manipulation failure, and the paper does not measure those stages separately.

Editorial extensions

If this is right

  • Free-form instructions become a sufficient interface for open-vocabulary mobile manipulation: the robot can zero-shot attempt tasks without per-task goal images or new demonstrations.
  • A single multi-task policy trained on 8 seen tasks with 100 demonstrations each transfers to 8 unseen OVMM tasks at 10.2% average success.
  • On unseen tabletop manipulation tasks, LOVMM's 12.9% average success at 100 demonstrations is more than three times the CLIPort baseline and more than ten times the 6-DoF Transporter baseline.
  • The 6-DoF pose module is doing real work: removing it and using only 3-DoF manipulation drops seen-task average success by 5.5 percentage points in the ablation.
  • Data augmentation and cropping the feature embedding rather than the input observation are necessary design choices; removing augmentation alone costs 19.1 percentage points of seen-task average success.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because navigation and parsing are inherited wholesale from pretrained components, the framework's genuinely trainable contribution is the 6-DoF manipulation policy; swapping in a different open-vocabulary navigation module or parser should leave the manipulation stage largely unchanged.
  • The absolute numbers are low (10.2% on unseen OVMM tasks), so if the claim is right the practical takeaway is that composition works but each stage remains a bottleneck; improving the weakest stages could raise end-to-end success more than further architecture changes.
  • A direct way to test the framework's claim is to report navigation and localization success separately, since the 10.2% figure is an upper bound on manipulation-only success and the paper does not measure those stages in isolation.
  • The benchmark uses only two scenes for seen tasks and eight for unseen tasks, all from the same simulator distribution; evaluating in more scenes or on a physical robot would show whether the zero-shot claim extends beyond the benchmark's distribution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LOVMM, a framework for language-conditioned open-vocabulary mobile manipulation that decomposes a free-form instruction into a navigation subtask and a manipulation subtask. The navigation part uses GPT-4 for parsing and VLMaps for open-vocabulary localization, while the manipulation part is a two-stream end-to-end model built on CLIPort-like semantics and Transporter-like spatial features, extended to predict 6-DoF poses. The authors construct 16 OVMM tasks (8 seen, 8 unseen) in Matterport3D scenes using the Habitat simulator, report task success rates, and additionally compare the manipulation module against Transporter6DoF and CLIPort on tabletop versions of the same tasks. The main results are a 10.2% average TSR on unseen OVMM tasks (Table 2) and a 12.9% average TSR for LOVMM versus 3.6% for CLIPort and 1.1% for Transporter6DoF on the tabletop tasks at 100 demonstrations (Table 5). The paper claims strong zero-shot generalization and superior multi-task learning compared to recent vision-based manipulation models.

Significance. If the claims are supported, the paper makes a useful contribution: a concrete modular architecture for OVMM that combines an LLM, a vision-language map, and a two-stream manipulation network, together with a new multi-task benchmark in complex home scenes. The manipulation model is clearly specified, the tabletop comparisons show a consistent advantage over CLIPort and Transporter6DoF, and the paper states that source code, dataset, and supplementary material are available, which would facilitate reproducibility. The work is a reasonable step toward practical OVMM. However, the headline claim of strong zero-shot generalization is strained by the low absolute unseen-task TSR, and the missing evaluation of the navigation/localization stage prevents attribution of the end-to-end results to the manipulation model. The significance would be substantially enhanced by component-level metrics and statistical rigor.

major comments (4)
  1. [Section 4.1 / Table 2] The central claim of strong zero-shot generalization rests on a 10.2% average TSR over eight unseen tasks. This number has no reported variance: there are no error bars, no standard deviations, and no repeated-seed runs, and the per-task values range from 3.2% to 21.2%. Without variance estimates, the reader cannot judge whether the differences between tasks, or the superiority over baselines, are statistically meaningful. Please report mean and standard deviation over at least three random seeds, and state whether the reported TSR is a single run or an average.
  2. [Section 3.2 / Section 4.1 / Table 6] The end-to-end TSR is the product of four stages: GPT-4 instruction parsing, VLMaps open-vocabulary localization, navigation to the workspace, and the learned manipulation policy. The paper never measures the success rate of the navigation/localization stage in the eight unseen scenes. Table 6 gives the baselines manually annotated fixed navigation routes (FNR) but does not include a LOVMM+FNR arm, so it cannot reveal how much of LOVMM's 10.2% unseen TSR is lost to navigation rather than manipulation. If navigation fails in a large fraction of episodes, the manipulation component's workspace-level success could be substantially higher than 10.2%, and the conclusion that the manipulation model drives the result would be unsupported. Please report navigation/localization success rates on the unseen scenes, or add a LOVMM+FNR condition to decompose the error. The Limitations section (Appendix A.5) mentions simplified settings but does not acknowledge this missing metric.
  3. [Section 4.2 / Table 5] The tabletop experiments remove navigation entirely by giving all models the same workspace observation, so they validate the manipulation model in isolation. They do not support the mobile part of the OVMM claim. The text should state explicitly that these results are not evidence about open-vocabulary navigation or cross-workspace behavior, and should frame them as a component-level evaluation. Currently, the reader could infer that the tabletop results confirm the full OVMM pipeline, which they do not.
  4. [Section 1 (Contributions) and Section 4.1] The claim that LOVMM 'zero-shot complete[s] diverse OVMM tasks decently' is stronger than the data support: the unseen-task TSR is 10.2% on average, with the best task at 21.2% and three tasks below 4%. While the result is non-trivial for a challenging benchmark, calling it 'strong' is an overstatement. I recommend reframing the claim as 'non-trivial zero-shot generalization' and discussing the failure modes and bottlenecks, rather than arguing from the highest per-task numbers.
minor comments (5)
  1. [Abstract] There is a typo: 'open- V ocabulary' should be 'open-vocabulary'.
  2. [Section 4.1, discussion after Table 1] The sentence 'LOVMM trained with 100 task demonstrations outperforms other models in half of the tasks' is unsupported as written, because Table 1 only contains LOVMM results. If the comparison refers to Table 5 or another table, please move or clarify the reference.
  3. [Section 3.3] The hyperparameters c = 64, k = 36, and d = 3 are introduced without defining d as the feature-channel dimension. The text later uses d' = 24; please define both clearly and explain the relationship.
  4. [Section 4.1 / Appendix A.1] The paper states that the dataset includes 'over 35K steps of demonstrations' and then says models are trained with n = 1, 10, 100 expert demonstrations. Please clarify how the 35K steps relates to the per-task demonstration counts and how demonstrations are generated or sampled.
  5. [Appendix A.5] The Limitations section mentions simplified task settings but does not mention the absence of navigation success metrics or the lack of error bars. Adding these points would make the limitations more complete and align with the evidence presented.

Circularity Check

0 steps flagged · score 0.0 of 10

No load-bearing circularity; the empirical claims rest on unseen-task evaluation against external components and benchmarks.

full rationale

LOVMM's central claims are empirical and are evaluated against tasks and baselines that are not equivalent to the paper's own fitted values. The end-to-end manipulation policy (Eqs. 3-9) is trained by imitation on seen-task demonstrations and then evaluated on unseen tasks; the unseen-task TSR (Table 2) and the tabletop comparisons (Table 5) are measurements, not quantities equal by construction to the training inputs. The navigation component in Section 3.2 is explicitly inherited from VLMaps [Huang et al., 2023], and the instruction parser is GPT-4, both external cited components; the reference list contains no author self-citations, so no self-citation chain is load-bearing. The strongest review concern is attribution rather than circularity: the mobile OVMM numbers in Table 2 are never decomposed by stage, so navigation or localization failures are conflated with manipulation failures, and Appendix A.3 deliberately removes navigation for the tabletop comparison. This weakens the claim that the manipulation model alone explains the 10.2% average TSR, but it does not make any equation or fitted parameter identical to the reported result. The reuse of CLIPort/Transporter both as architectural components and as baselines is a disclosed, potentially favorable design choice, but it is not a circular derivation. Appendix A.5 discloses simplified settings but does not address the missing navigation metrics; that is an unaddressed correctness gap, not a circular step. No step in the claimed derivation reduces to its own inputs by construction, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests on pretrained models and simulator fidelity: CLIP/CLIP-like embeddings for semantic matching, VLMaps for open-vocabulary localization, GPT-4 for parsing, and Habitat/Matterport3D for scene physics. No new physical entities are introduced; the only hand-chosen constants are architecture hyperparameters and undisclosed learned weights.

free parameters (4)
  • yaw rotation bins k = 36
    Hand-chosen discrete angle count for planar place pose in Eq. (5); no sensitivity analysis is reported.
  • pick feature crop size c = 64
    Hand-chosen crop size for the query template in Eq. (4); the paper ablates whether to crop the observation or the embedding, but not the crop size.
  • feature channels d' = 24
    Hand-chosen channel dimension after 1x1 convolution, split into three subsets for roll, pitch, and height regression in Eq. (6) without tuning analysis.
  • training steps = 600K
    Hand-chosen training duration across all seen tasks; convergence is not analyzed.
assumptions (6)
  • domain assumption Frozen CLIP image and text encoders provide a semantic space in which text descriptions of novel household objects align with pixels well enough to condition manipulation.
    Used in Eq. (7) and in VLMaps; the entire open-vocabulary claim relies on this alignment, but no standalone CLIP retrieval accuracy on the object set is reported.
  • domain assumption VLMaps constructed from LSeg pixel embeddings localize target workspaces in the simulator with sufficient accuracy for navigation.
    Section 3.2 directly inherits [Huang et al. 2023] and Eq. (2); localization success is not measured in the 8 unseen scenes.
  • domain assumption GPT-4 reliably parses free-form instructions into a target workspace and a target manipulation description.
    Section 3.2 uses GPT-4 as the parser; no prompt, parsing accuracy, or failure analysis is reported.
  • domain assumption Habitat/Matterport3D/Google Scanned Objects simulation is a faithful proxy for real household mobile manipulation.
    All experiments are in simulation; the conclusion's real-world applicability claims extrapolate beyond the evidence.
  • domain assumption The expert demonstrations and TSR metrics correctly capture the intended tasks.
    Demonstration generation is inherited from CLIPort-style scripts; no human validation of the 35K demonstration steps is described.
  • domain assumption A two-stream FCN trained with cross-entropy and Huber losses can learn 6-DoF pick-and-place from top-down RGB-D crops.
    This is the modeling premise behind Eqs. (3)-(6) and is only tested via aggregate success rates, not component analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Language-Conditioned Open-Vocabulary Mobile Manipulation with Pretrained Models." pith.science (2026). https://pith.science/paper/SXMGO4WU

@misc{pith2026250717379,
  author       = {Pith},
  title        = {Pith review of: Language-Conditioned Open-Vocabulary Mobile Manipulation with Pretrained Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SXMGO4WU}},
  note         = {Machine review of arXiv:2507.17379}
}
read the original abstract

Open-vocabulary mobile manipulation (OVMM) that involves the handling of novel and unseen objects across different workspaces remains a significant challenge for real-world robotic applications. In this paper, we propose a novel Language-conditioned Open-Vocabulary Mobile Manipulation framework, named LOVMM, incorporating the large language model (LLM) and vision-language model (VLM) to tackle various mobile manipulation tasks in household environments. Our approach is capable of solving various OVMM tasks with free-form natural language instructions (e.g. "toss the food boxes on the office room desk to the trash bin in the corner", and "pack the bottles from the bed to the box in the guestroom"). Extensive experiments simulated in complex household environments show strong zero-shot generalization and multi-task learning abilities of LOVMM. Moreover, our approach can also generalize to multiple tabletop manipulation tasks and achieve better success rates compared to other state-of-the-art methods.

Figures

Figures reproduced from arXiv: 2507.17379 by the authors.

Figure 1
Figure 1. Natural language-conditioned unseen OVMM tasks. We conduct large-scale experiments based on the CLIPort benchmark in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of LOVMM. We decompose OVMM as a series of open-vocabulary robot navigation and manipulation subtasks. Given a [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Architecture of the proposed end-to-end manipulation model, which adopts a two-stream architecture to fuse the visual obser [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Average task success rates for tabletop manipulation tasks. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Natural language-conditioned seen OVMM tasks. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Scenes for OVMM tasks. Livingroom-Carpet-Bed-Pack-Shapes-6dof (Task-G) This task is identical to livingroom-carpet-bed-pack-shapes, except that the target box is initialized with a random fixed 6- DoF pose within the angular range of ± π 6 and height limit of 10cm, whi…
Figure 7
Figure 7. Figure 7: Objects for OVMM tasks [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Data augmentation for training observation samples. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Bounding Boxes as Goals: Language-Conditioned Grasping via Neuro-Symbolic Planning

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    GRASP maps natural language to bounding-box goals via VLM for neuro-symbolic planning and reports 73.3% success in 90 real-robot trials without task-specific training.

  2. Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactions

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    POLAR organizes prior interactions into a multimodal knowledge graph with semantic and episodic memory to improve personalized embodied task execution across multiple MLLM backbones.

Reference graph

Works this paper leans on

42 extracted references · 30 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gpt-4 technical report

    [Achiam et al., 2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Alt- man, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv 2023

  2. [4]

    Mat- terport3d: Learning from rgb-d data in indoor environ- ments

    [Chang et al., 2017] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Mat- terport3d: Learning from rgb-d data in indoor environ- ments. arXiv preprint arXiv:1709.06158,

    arXiv 2017

  3. [5]

    It shows that LOVMM outperforms all the other models in 21/24 = 87 .5% of the evaluated tasks. Specifi- cally, LOVMM performs exceptionally well compared with the baselines in many tasks, achieving a best 55.9% TSR for bed-table-organize-bottles with only 1 demonstration, and a 26.6% performance for desk-corner-tidy-food. For tasks that are more challengi...

    work page 2022

  4. [6]

    Cows on pasture: Baselines and benchmarks for language- driven zero-shot object navigation

    [Gadre et al., 2023] Samir Yitzhak Gadre, Mitchell Worts- man, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song. Cows on pasture: Baselines and benchmarks for language- driven zero-shot object navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23171–23181,

    work page 2023

  5. [7]

    DiCarlo, Josh Mc- Dermott, and Antonio Torralba

    [Gan et al., 2022] Chuang Gan, Siyuan Zhou, Jeremy Schwartz, Seth Alter, Abhishek Bhandwaldar, Dan Gut- freund, Daniel LK Yamins, James J. DiCarlo, Josh Mc- Dermott, and Antonio Torralba. The threedworld transport challenge: A visually guided task-and-motion planning benchmark towards physically realistic embodied ai. In 2022 International Conference on R...

    work page 2022

  6. [8]

    Rlafford: End-to-end affordance learning for robotic manipulation

    [Geng et al., 2023b] Yiran Geng, Boshi An, Haoran Geng, Yuanpei Chen, Yaodong Yang, and Hao Dong. Rlafford: End-to-end affordance learning for robotic manipulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5880–5886. IEEE,

    work page 2023

  7. [9]

    Rvt: Robotic view transformer for 3d object manipulation

    [Goyal et al., 2023] Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox. Rvt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning, pages 694–710. PMLR,

    work page 2023

  8. [10]

    Rvt-2: Learn- ing precise manipulation from few demonstrations

    [Goyal et al., 2024] Ankit Goyal, Valts Blukis, Jie Xu, Yi- jie Guo, Yu-Wei Chao, and Dieter Fox. Rvt-2: Learn- ing precise manipulation from few demonstrations. arXiv preprint arXiv:2406.08545,

    arXiv 2024

Show all 42 references
  1. [12]

    Visual Language Maps for Robot Navigation

    [Huang et al., 2023] Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual Language Maps for Robot Navigation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 10608–10615, May

    2023

  2. [13]

    Mdetr-modulated detection for end-to-end multi- modal understanding

    [Kamath et al., 2021] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi- modal understanding. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 1780– 1790,

    2021

  3. [15]

    Openvla: An open-source vision-language- action model

    [Kim et al., 2024] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag San- keti, et al. Openvla: An open-source vision-language- action model. arXiv preprint arXiv:2406.09246,

    2024 arXiv

  4. [16]

    Berg, and Wan-Yen Lo

    [Kirillov et al., 2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, and Wan-Yen Lo. Segment anything. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pa...

    2023

  5. [17]

    Language-driven semantic segmentation

    [Li et al., 2022] Boyi Li, Kilian Q Weinberger, Serge Be- longie, Vladlen Koltun, and Ren´e Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201.03546,

    2022 arXiv

  6. [18]

    Grounding dino: Mar- rying dino with grounded pre-training for open-set object detection

    [Liu et al., 2023] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Mar- rying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499,

    2023 arXiv

  7. [19]

    Moka: Open-vocabulary robotic ma- nipulation through mark-based visual prompting

    [Liu et al., 2024] Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-vocabulary robotic ma- nipulation through mark-based visual prompting. In First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024 ,

    2024

  8. [20]

    Tax-pose: Task-specific cross-pose estimation for robot manipulation

    [Pan et al., 2023] Chuer Pan, Brian Okorn, Harry Zhang, Ben Eisner, and David Held. Tax-pose: Task-specific cross-pose estimation for robot manipulation. In Confer- ence on Robot Learning , pages 1783–1792. PMLR,

    2023

  9. [21]

    Open-vocabulary mobile manipulation in unseen dynamic environments with 3d se- mantic maps

    [Qiu et al., 2024] Dicong Qiu, Wenzong Ma, Zhenfu Pan, Hui Xiong, and Junwei Liang. Open-vocabulary mobile manipulation in unseen dynamic environments with 3d se- mantic maps. arXiv preprint arXiv:2406.18115,

    2024 arXiv

  10. [22]

    Learning transferable visual models from natural language supervision

    [Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, and Jack Clark. Learning transferable visual models from natural language supervision. In International Conference on ...

    2021

  11. [23]

    Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned trans- porter networks

    [Seita et al., 2021] Daniel Seita, Pete Florence, Jonathan Tompson, Erwin Coumans, Vikas Sindhwani, Ken Gold- berg, and Andy Zeng. Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned trans- porter networks. In 2021 IEEE International Conference on ...

    2021

  12. [24]

    Correcting robot plans with natural language feedback.arXiv preprint arXiv:2204.05186,

    [Sharma et al., 2022] Pratyusha Sharma, Balakumar Sundar- alingam, Valts Blukis, Chris Paxton, Tucker Hermans, An- tonio Torralba, Jacob Andreas, and Dieter Fox. Correcting robot plans with natural language feedback.arXiv preprint arXiv:2204.05186,

    2022 arXiv

  13. [25]

    Cliport: What and where pathways for robotic manipulation

    [Shridhar et al., 2022] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning , pages 894–906. PMLR,

    2022

  14. [26]

    Perceiver-actor: A multi-task transformer for robotic manipulation

    [Shridhar et al., 2023] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning , pages 785–799. PMLR,

    2023

  15. [27]

    Open-world object manipulation us- ing pre-trained vision-language models

    [Stone et al., 2023] Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, et al. Open-world object manipulation us- ing pre-trained vision-language models. arXiv preprint arXiv:2303.00905,

    2023 arXiv

  16. [28]

    Habitat 2.0: Training home assistants to rearrange their habitat

    [Szot et al., 2021] Andrew Szot, Alexander Clegg, Eric Un- dersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, and Oleksandr Maksymets. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in neural info...

    2021

  17. [29]

    Octo: An open-source generalist robot policy

    [Team et al., 2024] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213,

    2024 arXiv

  18. [30]

    TidyBot: Personalized robot assistance with large language models

    [Wu et al., 2023] Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. TidyBot: Personalized robot assistance with large language models. Autonomous Robots, 47(8):1087–1102, December

    2023

  19. [31]

    The best of both modes: Sepa- rately leveraging rgb and depth for unseen object instance segmentation

    [Xie et al., 2020] Christopher Xie, Yu Xiang, Arsalan Mousavian, and Dieter Fox. The best of both modes: Sepa- rately leveraging rgb and depth for unseen object instance segmentation. In Conference on Robot Learning , pages 1369–1378. PMLR,

    2020

  20. [33]

    A quantifiable stratification strategy for tidy-up in service robotics

    [Yan et al., 2021] Zhi Yan, Nathan Crombez, Jocelyn Buis- son, Yassine Ruichck, Tomas Krajnik, and Li Sun. A quantifiable stratification strategy for tidy-up in service robotics. In 2021 IEEE International Conference on Ad- vanced Robotics and Its Social Impacts (ARSO) , pages...

    2021

  21. [34]

    Depth anything: Unleashing the power of large-scale unlabeled data

    [Yang et al., 2024] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10371– 10381,

    2024

  22. [35]

    Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yang, Vidhi Jain, Alexander Clegg, John M

    [Yenamandra et al., 2023] Sriram Yenamandra, Arun Ra- machandran, Karmesh Yadav, Austin S. Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yang, Vidhi Jain, Alexander Clegg, John M. Turner, Zsolt Kira, Manolis Savva, Angel X. Chang, Devendra Singh Chaplot, Dhruv Batra, Roozbeh...

    2023

  23. [36]

    Transporter networks: Rearranging the visual world for robotic manipulation

    [Zeng et al., 2021] Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attar- ian, Travis Armstrong, Ivan Krasin, Dan Duong, and Vikas Sindhwani. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot L...

    2021

  24. [37]

    [Zeng et al., 2022] Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois R. Hogan, Maria Bauza, Daolin Ma, Orion Taylor, Melody Liu, Eudald Romo, Nima Fazeli, Ferran Alet, Nikhil Chavan Dafle, Rachel Holla- day, Isabella Morona, Prem Qu Nair, Druck Green, Ian Taylor,...

    2022

  25. [38]

    Supplementary Material for LOVMM A Appendix A.1 Natural Language-conditioned OVMM Task Details We construct our natural language-conditioned OVMM tasks by extending the CLIPort benchmark [Shridhar et al., 2022] into 10 different indoor scenes from the Matterport3D dataset [Cha...

    2022

  26. [39]

    To evaluate the multi-task learning and zero-shot generalizing capabilities of our model, we build the seen tasks based on 2 simple scenes that only in- volve 4 workspaces with clean backgrounds such as the top of a flat sofa, while using 8 completely different scenes with 12 ...

    2022

  27. [40]

    pack all the gray blocks on the bedroom sofa to the brown box

    For each task demon- stration, the mobile robot is initialized in a random position in the scene. We name all the tasks with the corresponding scenes, target workspaces, and target manipulation descrip- tions. Unlike the original fixed workspace settings, we add an additional ...

    2022

  28. [42]

    As a result, the dataset is heav- ily imbalanced

    A.5 Limitations Imbalanced Dataset Our proposed natural language-conditioned OVMM tasks in- volve a wide range of scenarios, covering single-step manip- ulation to long-horizon tasks. As a result, the dataset is heav- ily imbalanced. In order to have a fair and direct comparis...

    2024

  29. [2017]

    McHugh, and Vincent Vanhoucke

    [Downs et al., 2022] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Rey- mann, Thomas B. McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Confer- ence on Robotics...

    2022

  30. [2020]

    ULIP: Learning a unified representation of language, images, and point clouds for 3D understanding

    [Xue et al., 2023] Le Xue, Mingfei Gao, Chen Xing, Roberto Mart´ın-Mart´ın, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. ULIP: Learning a unified representation of language, images, and point clouds for 3D understanding. In Proceedings of the IEE...

    2023

  31. [2021]

    Interact: Trans- former models for human intent prediction conditioned on robot actions

    [Kedia et al., 2024] Kushal Kedia, Atiksh Bhardwaj, Prith- wish Dan, and Sanjiban Choudhury. Interact: Trans- former models for human intent prediction conditioned on robot actions. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 621–628. IEEE,

    2024

  32. [2022]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    [Brohan et al., 2023] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint a...

    2023 arXiv

  33. [2023]

    Do as i can, not as i say: Grounding language in robotic affordances.arXiv preprint arXiv:2204.01691,

    [Ahn et al., 2022] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances.arXiv preprint arXiv:2204.01691,

    2022 arXiv

  34. [2024]

    Adrian, Joshua Weil, Miroslav Gabriel, Philipp Schillinger, Markus Spies, Heiko Neumann, and Andras Gabor Kupcsik

    [Graf et al., 2023] Christian Graf, David B. Adrian, Joshua Weil, Miroslav Gabriel, Philipp Schillinger, Markus Spies, Heiko Neumann, and Andras Gabor Kupcsik. Learning dense visual descriptors using image augmentations for robot manipulation tasks. In Conference on Robot Lear...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.