RECORD
Learning Hierarchical Interactive Multi-Object Search for Mobile Manipulation
Robotics · Submitted 12 Jul 2023
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim We perform extensive experiments in simulation and the real world that demonstrate that, with accurate perception, the decision making of HIMOS effectively transfers to new environments in a zero-shot manner.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Existing object-search approaches enable robots to search through free pathways, however, robots operating in unstructured human-centered environments frequently also have to manipulate the environment to their needs. In this work, we introduce a novel interactive multi-object search task in which a robot has to open doors to navigate rooms and search inside cabinets and drawers to find target objects. These new challenges require combining manipulation and navigation skills in unexplored environments. We present HIMOS, a hierarchical reinforcement learning approach that learns to compose exploration, navigation, and manipulation skills. To achieve this, we design an abstract high-level action space around a semantic map memory and leverage the explored environment as instance navigation points. We perform extensive experiments in simulation and the real world that demonstrate that, with accurate perception, the decision making of HIMOS effectively transfers to new environments in a zero-shot manner. It shows robustness to unseen subpolicies, failures in their execution, and different robot kinematics. These capabilities open the door to a wide range of downstream tasks across embodied AI and real-world use cases.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Existing object-search approaches enable robots to search through free pathways, however, robots operating in unstructured human-centered environments frequently also have to manipulate the environment to their needs. | background 0.77 | 0.04 | 0.02 | asserted 0.99 |
| s1 | In this work, we introduce a novel interactive multi-object search task in which a robot has to open doors to navigate rooms and search inside cabinets and drawers to find target objects. | contribution 0.99 | 0.56 | 0.05 | asserted 0.97 |
| s2 | These new challenges require combining manipulation and navigation skills in unexplored environments. | background 0.54 | 0.09 | 0.04 | asserted 0.99 |
| s3 | We present HIMOS, a hierarchical reinforcement learning approach that learns to compose exploration, navigation, and manipulation skills. | contribution 0.85 | 0.94 | 0.11 | asserted 1.00 |
| s4 | To achieve this, we design an abstract high-level action space around a semantic map memory and leverage the explored environment as instance navigation points. | method 0.99 | 0.92 | 0.04 | asserted 0.99 |
| s5 | We perform extensive experiments in simulation and the real world that demonstrate that, with accurate perception, the decision making of HIMOS effectively transfers to new environments in a zero-shot manner. | result 1.00 | 0.97 | 0.57 | asserted 1.00 |
| s6 | It shows robustness to unseen subpolicies, failures in their execution, and different robot kinematics. | result 0.99 | 0.89 | 0.12 | asserted 1.00 |
| s7 | These capabilities open the door to a wide range of downstream tasks across embodied AI and real-world use cases. | interpretation 0.94 | 0.23 | 0.22 | asserted 0.54 |
What was examined, and what was not
- arXiv listing abstract v3
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Existing object-search approaches enable robots to search through free pathways, however, robots operating in unstructured human-centered environments frequently also have to manipulate the environment to their needs. In this work, we introduce a novel interactive multi-object search task in which a robot has to open doors to navigate rooms and search inside cabinets and drawers to find target objects. These new challenges require combining manipulation and navigation skills in unexplored environments. We present HIMOS, a hierarchical reinforcement learning approach that learns to compose exploration, navigation, and manipulation skills. To achieve this, we design an abstract high-level action space around a semantic map memory and leverage the explored environment as instance navigation points. We perform extensive experiments in simulation and the real world that demonstrate that, with accurate perception, the decision making of HIMOS effectively transfers to new environments in a zero-shot manner. It shows robustness to unseen subpolicies, failures in their execution, and different robot kinematics. These capabilities open the door to a wide range of downstream tasks across embodied AI and real-world use cases.
Human review
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites
Compares with 1 paper
-
In this section, we compare with recent work on semantic object-navigation [ref] .
ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation · Ablation: Geometric Utility Baseline · 1.00
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.