REVIEW 5 cited by
Policy Expansion for Bridging Offline-to-Online Reinforcement Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Pre-training with offline data and online fine-tuning using reinforcement learning is a promising strategy for learning control policies by leveraging the best of both worlds in terms of sample efficiency and performance. One natural approach is to initialize the policy for online learning with the one trained offline. In this work, we introduce a policy expansion scheme for this task. After learning the offline policy, we use it as one candidate policy in a policy set. We then expand the policy set with another policy which will be responsible for further learning. The two policies will be composed in an adaptive manner for interacting with the environment. With this approach, the policy previously learned offline is fully retained during online learning, thus mitigating the potential issues such as destroying the useful behaviors of the offline policy in the initial stage of online learning while allowing the offline policy participate in the exploration naturally in an adaptive manner. Moreover, new useful behaviors can potentially be captured by the newly added policy through learning. Experiments are conducted on a number of tasks and the results demonstrate the effectiveness of the proposed approach. Code is available at https://github.com/Haichao-Zhang/PEX
Forward citations
Cited by 5 Pith papers
-
Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?
On six robot-manipulation tasks, offline Q-pretraining does not accelerate online RL fine-tuning from a pretrained policy, while seeding the replay buffer with rollouts from an ensemble of policies (IPE) improves fina...
-
Behavioral Exploration: Learning to Explore via In-Context Adaptation
A coverage-conditioned behavioral cloning policy adapts in-context to its own history, making robots explore new expert-like behaviors online without online reinforcement learning.
-
Online Pre-Training for Offline-to-Online Reinforcement Learning
A new 'online pre-training' phase trains a second value function that is then blended with the offline one during fine-tuning, improving offline-to-online RL across D4RL benchmarks.
-
Reinforcement Learning via Implicit Imitation Guidance
A reinforcement learning method that learns a state-dependent covariance from expert-policy action differences and uses it as exploration noise, improving sample efficiency on sparse-reward continuous control tasks.
-
Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
SSAR replaces the fixed global regularization strength in offline RL with state-adaptive coefficients and applies regularization only to high-quality actions, improving D4RL performance over CQL and TD3+BC.
Discussion (0). Sign in to comment.