RECORD
Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space
Robotics · Submitted 17 May 2022
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Our experimental results show that PTP can generate feasible sequences of subgoals that enable the policy to efficiently solve the target tasks.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
General-purpose robots require diverse repertoires of behaviors to complete challenging tasks in real-world unstructured environments. To address this issue, goal-conditioned reinforcement learning aims to acquire policies that can reach configurable goals for a wide range of tasks on command. However, such goal-conditioned policies are notoriously difficult and time-consuming to train from scratch. In this paper, we propose Planning to Practice (PTP), a method that makes it practical to train goal-conditioned policies for long-horizon tasks that require multiple distinct types of interactions to solve. Our approach is based on two key ideas. First, we decompose the goal-reaching problem hierarchically, with a high-level planner that sets intermediate subgoals using conditional subgoal generators in the latent space for a low-level model-free policy. Second, we propose a hybrid approach which first pre-trains both the conditional subgoal generator and the policy on previously collected data through offline reinforcement learning, and then fine-tunes the policy via online exploration. This fine-tuning process is itself facilitated by the planned subgoals, which breaks down the original target task into short-horizon goal-reaching tasks that are significantly easier to learn. We conduct experiments in both the simulation and real world, in which the policy is pre-trained on demonstrations of short primitive behaviors and fine-tuned for temporally extended tasks that are unseen in the offline data. Our experimental results show that PTP can generate feasible sequences of subgoals that enable the policy to efficiently solve the target tasks.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | General-purpose robots require diverse repertoires of behaviors to complete challenging tasks in real-world unstructured environments. | background 1.00 | 0.05 | 0.02 | asserted 1.00 |
| s1 | To address this issue, goal-conditioned reinforcement learning aims to acquire policies that can reach configurable goals for a wide range of tasks on command. | background 1.00 | 0.04 | 0.02 | asserted 0.98 |
| s2 | However, such goal-conditioned policies are notoriously difficult and time-consuming to train from scratch. | background 0.53 | 0.05 | 0.02 | asserted 1.00 |
| s3 | In this paper, we propose Planning to Practice (PTP), a method that makes it practical to train goal-conditioned policies for long-horizon tasks that require multiple distinct types of interactions to solve. | contribution 0.99 | 0.75 | 0.21 | asserted 1.00 |
| s4 | Our approach is based on two key ideas. | method 0.52 | 0.31 | 0.02 | asserted 0.55 |
| s5 | First, we decompose the goal-reaching problem hierarchically, with a high-level planner that sets intermediate subgoals using conditional subgoal generators in the latent space for a low-level model-free policy. | method 0.94 | 0.86 | 0.04 | asserted 0.99 |
| s6 | Second, we propose a hybrid approach which first pre-trains both the conditional subgoal generator and the policy on previously collected data through offline reinforcement learning, and then fine-tunes the policy via online exploration. | method 0.94 | 0.85 | 0.05 | asserted 1.00 |
| s7 | This fine-tuning process is itself facilitated by the planned subgoals, which breaks down the original target task into short-horizon goal-reaching tasks that are significantly easier to learn. | method 0.97 | 0.52 | 0.14 | asserted 1.00 |
| s8 | We conduct experiments in both the simulation and real world, in which the policy is pre-trained on demonstrations of short primitive behaviors and fine-tuned for temporally extended tasks that are unseen in the offline data. | method 0.92 | 0.95 | 0.08 | asserted 0.97 |
| s9 | Our experimental results show that PTP can generate feasible sequences of subgoals that enable the policy to efficiently solve the target tasks. | result 0.99 | 0.97 | 0.76 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v2
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
General-purpose robots require diverse repertoires of behaviors to complete challenging tasks in real-world unstructured environments. To address this issue, goal-conditioned reinforcement learning aims to acquire policies that can reach configurable goals for a wide range of tasks on command. However, such goal-conditioned policies are notoriously difficult and time-consuming to train from scratch. In this paper, we propose Planning to Practice (PTP), a method that makes it practical to train goal-conditioned policies for long-horizon tasks that require multiple distinct types of interactions to solve. Our approach is based on two key ideas. First, we decompose the goal-reaching problem hierarchically, with a high-level planner that sets intermediate subgoals using conditional subgoal generators in the latent space for a low-level model-free policy. Second, we propose a hybrid approach which first pre-trains both the conditional subgoal generator and the policy on previously collected data through offline reinforcement learning, and then fine-tunes the policy via online exploration. This fine-tuning process is itself facilitated by the planned subgoals, which breaks down the original target task into short-horizon goal-reaching tasks that are significantly easier to learn. We conduct experiments in both the simulation and real world, in which the policy is pre-trained on demonstrations of short primitive behaviors and fine-tuned for temporally extended tasks that are unseen in the offline data. Our experimental results show that PTP can generate feasible sequences of subgoals that enable the policy to efficiently solve the target tasks.
Human review
References
How the literature uses it
5 arXiv papers cite this in 17 sentences
Papers citing it · 13 sentences without a confident label
Compares with it 1 paper
-
PTP [ref] focuses on finding sub-goals in a pre-trained VQ-VAE [ref] representation space and learns a policy to reach them sequentially.
Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data · Comparing to Prior Offline Goal-conditioned RL Methods · 0.89
Cites it as background 2 papers
-
Some have also studied offline RL followed by a small amount of policy fine-tuning [ref] with the goal of finding policies that optimize simple regret.
Bridging Imitation and Online Reinforcement Learning: An Optimistic Tale · Introduction · 0.84
-
Prior works has proposed to combine critic learning with an explicit planner [ref] , using the learned value network as distance function to generate sub-goals.
Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data · Sub-goal planning · 0.85
What it cites · 1 sentence without a confident label
Uses the method or tool of 1 paper
-
Implicit Q-Learning (IQL) [ref] is used as the underlying RL algorithm for offline pre-training and online fine-tuning with default hyperparameters.
Offline Reinforcement Learning with Implicit Q-Learning · Implementation Details · 1.00
Cites as background 2 papers
-
These prior methods either propose subgoals from the set of previously seen states, or directly optimize over subgoals, often by utilizing a latent variable model to obtain a concise representation of image-based states [ref] .
Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation · Related Work · 0.96
-
Offline reinforcement learning methods learn from prior data [ref] , and can also finetune through online interaction [ref] .
A Minimalist Approach to Offline Reinforcement Learning · Related Work · 0.94
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.