REVIEW 5 cited by
A Minimalist Approach to Offline Reinforcement Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Offline reinforcement learning (RL) defines the task of learning from a fixed batch of data. Due to errors in value estimation from out-of-distribution actions, most offline RL algorithms take the approach of constraining or regularizing the policy with the actions contained in the dataset. Built on pre-existing RL algorithms, modifications to make an RL algorithm work offline comes at the cost of additional complexity. Offline RL algorithms introduce new hyperparameters and often leverage secondary components such as generative models, while adjusting the underlying RL algorithm. In this paper we aim to make a deep RL algorithm work while making minimal changes. We find that we can match the performance of state-of-the-art offline RL algorithms by simply adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data. The resulting algorithm is a simple to implement and tune baseline, while more than halving the overall run time by removing the additional computational overhead of previous methods.
Forward citations
Cited by 5 Pith papers
-
Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
A taxonomy of robot learning on a weights-versus-skills axis, with a five-rung self-improvement ladder whose top cell (feedback plus memory plus search) holds only a few recent systems.
-
Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?
On six robot-manipulation tasks, offline Q-pretraining does not accelerate online RL fine-tuning from a pretrained policy, while seeding the replay buffer with rollouts from an ensemble of policies (IPE) improves fina...
-
Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation
CQ2L uses Morse-network uncertainty to select in-distribution action queries and to scale CQL's regularization, reporting higher D4RL scores than the prior OAP method.
-
FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning
FlowQ uses energy-guided flow matching to learn an offline RL policy approximating π(a|s) ∝ πβ(a|s) exp(Q(s,a)) with guidance applied during training rather than at inference.
-
Value Flows
Value Flows fits the full return distribution in RL with a flow-matching critic and reweights its learning objective by estimated return variance; the central theoretical guarantee does not follow from the stated equations.
Discussion (0). Sign in to comment.