REVIEW 27 cited by
OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Remarkable progress has been made in recent years in the fields of vision, language, and robotics. We now have vision models capable of recognizing objects based on language queries, navigation systems that can effectively control mobile systems, and grasping models that can handle a wide range of objects. Despite these advancements, general-purpose applications of robotics still lag behind, even though they rely on these fundamental capabilities of recognition, navigation, and grasping. In this paper, we adopt a systems-first approach to develop a new Open Knowledge-based robotics framework called OK-Robot. By combining Vision-Language Models (VLMs) for object detection, navigation primitives for movement, and grasping primitives for object manipulation, OK-Robot offers a integrated solution for pick-and-drop operations without requiring any training. To evaluate its performance, we run OK-Robot in 10 real-world home environments. The results demonstrate that OK-Robot achieves a 58.5% success rate in open-ended pick-and-drop tasks, representing a new state-of-the-art in Open Vocabulary Mobile Manipulation (OVMM) with nearly 1.8x the performance of prior work. On cleaner, uncluttered environments, OK-Robot's performance increases to 82%. However, the most important insight gained from OK-Robot is the critical role of nuanced details when combining Open Knowledge systems like VLMs with robotic modules. Videos of our experiments and code are available on our website: https://ok-robot.github.io
Forward citations
Cited by 27 Pith papers
-
What's in the Image? A Deep-Dive into the Vision of Vision Language Models
Vision-language models store a global image summary in the query text tokens, rely on the middle transformer layers for vision-to-text transfer, and fetch fine details from image tokens in a spatially localized way.
-
Aim My Robot: Precision Local Navigation to Any Object
AMR is an end-to-end vision-based local navigation system that reaches a desired relative pose to an object with centimeter-level precision, using a reference image plus mask and multi-modal sensing.
-
Deltoris: Enabling Real-time VLA Inference in Embodied AI via Bit-level Sparsity and Speculative Inference
Temporal-aware bit-level sparsity plus batched speculative verification lets a co-designed accelerator run diffusion-based vision-language-action models up to 34x faster than a mobile GPU.
-
HyperDCM: Dynamic Cluster Memory Replay in Hyperbolic Space for Continual Robotic Navigation Across Scenes
A scene-graph-plus-hyperbolic replay memory reduces reported performance drops in continual diffusion navigation, but the headline Drop metric is defined across different scenes and does not clearly measure forgetting.
-
TiPToP: A Modular Open-Vocabulary Robot Manipulation System That Plans
TiPToP, a zero-training modular planner using pretrained vision-language models and GPU-accelerated TAMP, achieves 74.6% success over 165 trials versus 52.4% for the 350-hour-trained pi0.5-DROID baseline across 28 man...
-
PLanAR: Planning-Language-Grounded Agentic Reasoning for Robot Manipulation
AgenticLab's closed-loop planning-language pipeline lets different vision-language models drive a real robot, and benchmark tests show action-verification quality, not planning, determines long-horizon success.
-
FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policies
A compact 950-million-parameter robot policy trained in about 200 GPU-hours matches or beats multi-billion-parameter baselines on most manipulation benchmarks, including a new best score on CALVIN ABC.
-
GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Training
GraspGen shows that training a grasp-scoring discriminator on the generator's own simulated outputs, plus a large new multi-gripper dataset, improves 6-DOF grasping across simulation and a real robot.
-
FineGrasp: Towards Robust Grasping for Delicate Objects
FineGrasp improves 6-DoF grasp detection for small/delicate objects using instance-normalized graspness labels, multi-range attention, surface-normal priors, and sim-to-real training.
-
SwitchVLA: Execution-Aware Task Switching for Vision-Language-Action Models
SwitchVLA trains a vision-language-action policy to handle mid-execution instruction changes by conditioning on contact state and a three-way behavior mode, using only existing single-task demonstrations.
-
Language-Guided Generation for Personalized Inspection Planning
A training-free pipeline uses a vision-language model and segmentation to convert text instructions into smooth, order-respecting drone inspection trajectories in known 3D maps.
-
Unfettered Forceful Skill Acquisition with Physical Reasoning and Coordinate Frame Labeling
Overlaying coordinate frames on robot camera images lets a general-purpose vision-language model generate executable wrench commands, achieving 51% zero-shot success on four manipulation tasks and revealing that embod...
-
MORE: Mobile Manipulation Rearrangement Through Grounded Language Reasoning
MORE uses LLM-based scene graph filtering and instance-level reasoning to solve 48.1% of 81 selected BEHAVIOR-1K rearrangement tasks, outperforming prior LLM and VLM planners in the same modified setup.
-
ImageInThat: Manipulating Images to Convey User Instructions to Robots
Users who directly manipulated images in the ImageInThat timeline interface generated robot instructions faster and with higher confidence than with a text-based method, in a ten-participant kitchen-task study.
-
WildLMa: Long Horizon Loco-Manipulation in the Wild
WildLMa combines VR teleoperation with whole-body control, CLIP-based language-conditioned imitation learning, and an LLM planner to give a quadruped robot reusable manipulation skills that generalize to unseen object...
-
ETA: A New Agentic Paradigm for Embodied Tasks
A general-purpose LLM planner using only observe, mark_point, and move_to solves 90% of 130 LIBERO manipulation tasks when allowed five attempts per task, with no robot-policy training.
-
N2M: Bridging Navigation and Manipulation by Learning Pose Preference from Rollout
N2M predicts preferable base poses for manipulation policies from ego-centric point clouds, learned from rollouts, lifting success from 3% to 54% in the PnPCounterToCab task.
-
AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulation
AC-DiT adds mobility-to-body conditioning and perception-aware 2D/3D weighting to a diffusion transformer, improving success rates on simulated and real-world mobile manipulation tasks.
-
MapleGrasp: Mask-guided Feature Pooling for Language-driven Efficient Robotic Grasping
A two-stage language-driven grasping system that pools visual features inside a predicted object mask improves grasp accuracy and training efficiency versus CLIP baselines, supported by a new 219M-grasp dataset.
-
OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis
A vision-language model fine-tuned on 572K synthetic simulation examples improves open-world mobile manipulation action decisions and object grounding over GPT-4o, with 21.9% full-task success in simulation and 90% ac...
-
BeliefMapNav: 3D Voxel-Based Belief Map for Zero-Shot Object Navigation
A zero-shot navigation system that builds a 3D voxel belief map from LLM-generated landmarks and CLIP features, then plans frontier paths by expected search distance, achieving state-of-the-art success rate and succes...
-
GLOVER++: Unleashing the Potential of Affordance Learning from Human Behaviors for Robotic Manipulation
GLOVER++ predicts where to interact with objects from language instructions after training on the new 500K-image HOVA-500K affordance dataset, and reports gains in simulated and real manipulation.
-
Can Large Language Models Help Developers with Robotic Finite State Machine Modification?
Two LLMs, given old FSM code and an automatically generated change summary, reproduced five of six real robot FSM modifications from a RoboCup team's git history.
-
GLOVER: Generalizable Open-Vocabulary Affordance Reasoning for Task-Oriented Grasping
GLOVER predicts open-vocabulary graspable regions on objects from one RGB image and estimates grasp poses from those regions, reporting large speedups and higher success rates than prior systems.
-
Kitchen Robotic Manipulation utilizing Foundation Models
A modular perception pipeline using off-the-shelf foundation models achieves 89.12% ADI on a custom kitchen dishware dataset and performs real robot sink-to-dishwasher and cup-stacking tasks without retraining.
-
GBPP: Grasp-Aware Base Placement Prediction for Robots via Two-Stage Learning
GBPP uses 180k cheap heuristic labels plus 12k simulation trials to train a point-cloud classifier that picks a mobile robot's base pose for grasping in about 0.3 seconds.
-
MoTo: A Zero-shot Plug-in Interaction-aware Navigation for General Mobile Manipulation
MoTo turns existing fixed-base manipulation models into mobile manipulators by using VLM-picked contact keypoints and trajectory optimization to find docking points, with no training of MoTo itself.
Discussion (0). Continue with ORCID to comment.