REVIEW 6 cited by
Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Autonomous robot navigation and manipulation in open environments require reasoning and replanning with closed-loop feedback. In this work, we present COME-robot, the first closed-loop robotic system utilizing the GPT-4V vision-language foundation model for open-ended reasoning and adaptive planning in real-world scenarios.COME-robot incorporates two key innovative modules: (i) a multi-level open-vocabulary perception and situated reasoning module that enables effective exploration of the 3D environment and target object identification using commonsense knowledge and situated information, and (ii) an iterative closed-loop feedback and restoration mechanism that verifies task feasibility, monitors execution success, and traces failure causes across different modules for robust failure recovery. Through comprehensive experiments involving 8 challenging real-world mobile and tabletop manipulation tasks, COME-robot demonstrates a significant improvement in task success rate (~35%) compared to state-of-the-art methods. We further conduct comprehensive analyses to elucidate how COME-robot's design facilitates failure recovery, free-form instruction following, and long-horizon task planning.
Forward citations
Cited by 6 Pith papers
-
Visual-Language-Guided Task Planning for Horticultural Robots
A vision-language model drives a simulated greenhouse robot through simple crop-inspection tasks with ~87% success, but long multi-target tasks collapse to under 10% success.
-
Robix: A Unified Model for Robot Interaction, Reasoning and Planning
A three-stage-trained VLM unifies robot planning and dialogue, and beats commercial VLMs on the authors' interactive-task benchmarks.
-
Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models
A VLM-powered assistive teleoperation system infers diverse user intents from teleoperation snippets and executes them with a skill library, outperforming baselines on real-world mobile manipulation tasks.
-
HERMES: Human-to-Robot Embodied Learning from Multi-Source Motion Data for Mobile Dexterous Manipulation
HERMES converts a single human motion demonstration into a deployable mobile bimanual dexterous manipulation policy, using RL, depth-image distillation, and closed-loop PnP pose refinement.
-
OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis
A vision-language model fine-tuned on 572K synthetic simulation examples improves open-world mobile manipulation action decisions and object grounding over GPT-4o, with 21.9% full-task success in simulation and 90% ac...
-
Gaze-supported Large Language Model Framework for Bi-directional Human-Robot Interaction
A gaze- and speech-driven LLM framework for assistive robots matches a scripted interaction pipeline on task performance while slightly increasing user-perceived confidence, at higher energy cost.
Discussion (0). Sign in to comment.